edgequake-llm

What It Covers

• One trait-based surface for LLMs, embeddings, and Rust image generation.
• Production backends: OpenAI, Azure OpenAI, Anthropic, Gemini, Vertex AI, xAI, OpenRouter, NVIDIA NIM, Mistral, AWS Bedrock.
• Local and gateway backends: Ollama, LM Studio, oMLX, MTPLX, llama-server (llamacpp), vLLM-MLX, mlx-lm (thin LocalOpenAiProvider shell), GitHub Copilot direct mode (proxy optional), generic OpenAI-compatible APIs.
• Additional embedding backend: Jina.
• Image generation backends in the Rust crate: Gemini image generation, Vertex Imagen, FAL, mock image generation.
• Model Discovery — programmatic capability discovery, provider catalog, and name/fuzzy search across 11+ providers with zero heuristics.
• Operational layers: caching, retry, rate limiting, cost tracking, tracing, reranking, mock providers.

Install

[dependencies]
edgequake-llm = "0.10.3"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

bedrock is feature-gated:

[dependencies]
edgequake-llm = { version = "0.10.3", features = ["bedrock"] }

Note: the repository is now pinned to Rust 1.95.0, and the Bedrock integration is verified against the latest published AWS SDK crate set, including the current Bedrock runtime release.

Provider compatibility highlights in this release:

• Anthropic-compatible adapters now preserve final streamed tool-call deltas even when the upstream SSE stream ends without a trailing newline.
• Provider-side schema normalization now aligns Anthropic, Bedrock, and Gemini tool declarations with the stricter subsets those APIs actually accept.
• Normalized usage reporting distinguishes cache writes from cache hits where the upstream provider reports both.
• Gemini provider now preserves function-call IDs across assistant tool calls, streamed deltas, and tool-result follow-ups.
• Mistral provider now includes native audio (speech, transcriptions, voices) and OCR endpoint wrappers.

New capabilities in v0.10.0:

• Application attribution — ApplicationContext, ProviderFactory::create_llm_provider_with_context(), Python ApplicationContext + application_id= kwargs, OTEL gen_ai.application.* spans and edgequake.attribution.* audit events, get_provider_attribution() catalog helpers
• Ollama Cloud — OllamaProvider::from_env_cloud(), auto https://ollama.com when OLLAMA_API_KEY is set; latest API (think, JSON schema, embed dimensions)
• Examples — application_attribution, ollama_cloud

New in v0.10.5 / v0.10.6 (SPEC-113 / #369):

• Capability-gated Ollama think — Auto no longer injects think: true from name substrings (qwen3, …). Truth is Ollama /api/show (and tags) capabilities including "thinking". Unknown/error → omit think.
• Env: EDGEQUAKE_OLLAMA_THINK_CAPABILITY (auto|force_off|force_on|legacy_name), EDGEQUAKE_OLLAMA_CAPABILITY_TTL_SECS, EDGEQUAKE_OLLAMA_CAPABILITY_TIMEOUT_MS
• v0.10.6: Unknown capability answers use a 5s TTL (not the full probe TTL); EDGEQUAKE_OLLAMA_THINK_CAPABILITY is read per request (not frozen at build()).

New capabilities in v0.9.0:

• Provider Catalog — ProviderFactory::list_providers(), ProviderCatalog::resolve_id(), unified metadata for chat/embed/discovery/image-gen surfaces
• Capability search — find_static_models(), ModelCapability enum, min/max input and output token bounds
• Model name search — exact, substring, fuzzy, and provider/model queries via ModelSearchQuery
• Python discovery — search_models(), search_static_models_by_name(), lookup_model_by_name()

Capabilities from v0.8.0 still included:

• Structured Outputs — CompletionOptions::json_schema(schema) for JSON Schema-constrained responses
• OpenRouter fallback models — with_fallback_models() for sequential model failover
• Azure Managed Identity — AzureCredential::BearerToken for Entra ID auth
• Vertex AI ADC — automatic GCE metadata token refresh
• Mistral variable embedding dimensions — with_embed_dimensions(dim)
• Ollama truncate parameter in embedding requests
• Bedrock Nova 2 multimodal embeddings
• NVIDIA vision model detection improvements

Latest model IDs validated on 2026-10-01 from official provider documentation:

• OpenAI (models): chat default gpt-5.6-terra (Chat Completions still supports tools). Catalog also includes gpt-6-astra, gpt-6.1-sol, gpt-6-luna, and the GPT-5.6 family. Image default gpt-image-2.5-flare.
• Anthropic (overview): default claude-sonnet-5-5; also claude-opus-5-5, claude-fable-5-1.
• Gemini (models): default gemini-3.8-flash. Image default gemini-3.1-flash-image.
• Mistral (models): default mistral-medium-3-5.
• xAI (models): default grok-4.7. Image default grok-imagine-image-2.0.
• Cohere (models): default command-a-plus-05-2026.

Quick Start

use edgequake_llm::{ChatMessage, LLMProvider, OpenAIProvider};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let provider = OpenAIProvider::from_env()?;
    let messages = vec![ChatMessage::user("Explain Rust ownership in one sentence.")];
    let response = provider.chat(&messages, None).await?;

    println!("{}", response.content);
    Ok(())
}

Environment:

export OPENAI_API_KEY=sk-...

Provider Matrix

Image Generation Providers

Rust-only image generation support is exposed through ImageGenProvider and ImageGenFactory:

Common Setup

GitHub Copilot direct mode

Use vscode-copilot/auto unless you have a strong reason to pin a specific model.

Why this is now the default:

• GitHub's live Auto routing knows which chat-capable model family is actually available for the current account and session.
• Some Copilot catalog entries are responses-only or temporarily throttled; Auto avoids hard-coding a brittle premium path.
• Reusing the real VS Code auth cache keeps parity with the official extension instead of depending on stale local token copies.

Legacy proxy setups still work through VSCODE_COPILOT_PROXY_URL, but no proxy is required for the normal path anymore.

Image generation environment:

Factory Usage

ProviderFactory is the fastest way to wire environments or provider/model routing:

use edgequake_llm::{ProviderFactory, ProviderType};

let (llm, embedding) = ProviderFactory::from_env()?;
println!("llm={} embedding={}", llm.name(), embedding.name());

let (vertex_llm, _) = ProviderFactory::create_with_model(
    ProviderType::VertexAI,
    Some("gemini-3.8-flash"),
)?;

let custom = ProviderFactory::create_llm_provider(
    "openai-compatible",
    "deepseek-chat",
)?;

For generic OpenAI-compatible routing, set:

export OPENAI_COMPATIBLE_BASE_URL=https://api.groq.com/openai/v1
export OPENAI_COMPATIBLE_API_KEY=...

For Rust image generation, use:

use edgequake_llm::{ImageGenFactory, ImageGenRequest};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let provider = ImageGenFactory::from_env()?;
    let response = provider
        .generate(&ImageGenRequest::new("Editorial product photo on a concrete desk"))
        .await?;

    println!("generated {} image(s)", response.images.len());
    Ok(())
}

Header Propagation

All five major providers support caller-supplied HTTP headers for B2B/multi-tenant deployments (closes edgequake#132):

Use with_extra_headers() to inject custom headers — trace IDs, tenant identifiers, HMAC tokens, or traceparent — into every outgoing LLM API call:

use edgequake_llm::{AnthropicProvider, GeminiProvider, MistralProvider};

// Anthropic
let provider = AnthropicProvider::from_env()?
    .with_extra_headers([
        ("x-request-id".to_string(), "req-abc-123".to_string()),
        ("x-tenant-id".to_string(), "tenant-42".to_string()),
    ]);

// Gemini (Google AI or VertexAI)
let provider = GeminiProvider::from_env()?
    .with_extra_headers([
        ("x-correlation-id".to_string(), "corr-xyz".to_string()),
    ]);

// Mistral
let provider = MistralProvider::from_env()?
    .with_extra_headers([
        ("traceparent".to_string(), "00-abc123-def456-01".to_string()),
    ]);

Reserved headers (authorization, x-api-key, anthropic-version, content-type, content-length, host, user-agent) are silently dropped to prevent accidental credential overrides. All other headers pass through to every request made by that provider instance.

Python Package

edgequake-litellm is the Python package in this repo. It is a drop-in LiteLLM replacement backed by the Rust runtime:

import edgequake_litellm as litellm

resp = litellm.completion(
    model="openai/gpt-5.6-terra",
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

Install:

pip install edgequake-litellm

See edgequake-litellm/README.md for provider routing, migration notes, wheel coverage, and release instructions. The Python package does not expose the Rust image-generation APIs yet.

Model Discovery

Programmatic model discovery across all providers — no hardcoded model IDs, no name-pattern heuristics.

List providers

use edgequake_llm::{ProviderCatalog, ProviderFactory};

let chat = ProviderFactory::list_providers();
let discovery = ProviderCatalog::list_discovery_providers();

import edgequake_litellm as litellm

print(litellm.list_providers())
print(litellm.discovery.list_discovery_providers())

Search by capability

use edgequake_llm::{find_static_models, CapabilityFilter, ModelCapability, ModelDiscoveryService};

// Offline — no API keys required
let filter = CapabilityFilter::default()
    .requiring(ModelCapability::Vision)
    .with_min_context_length(100_000)
    .with_min_output_tokens(8_192);
let models = find_static_models(&filter);

// Live discovery
let service = ModelDiscoveryService::new();
let live = service.find_models(&filter).await?;

Search by name (exact, substring, fuzzy)

use edgequake_llm::{search_static_models, ModelSearchQuery};

let hits = search_static_models(
    &ModelSearchQuery::new("claude sonnet")
        .fuzzy(true)
        .with_min_context_length(200_000)
        .with_min_output_tokens(16_384),
);
for hit in hits {
    println!("{} score={:.2}", hit.model.id, hit.score);
}

Python:

import edgequake_litellm as litellm

# Capability filter
models = litellm.discovery.find_static_models(
    requires_vision=True,
    min_context_length=100_000,
)

# Name / fuzzy search with length bounds
hits = litellm.discovery.search_static_models_by_name(
    "claude",
    fuzzy=True,
    min_context_length=200_000,
    min_output_tokens=16_384,
)
for hit in hits:
    print(f"{hit.model.provider}/{hit.model.id} score={hit.score:.2f}")

Features:

• Unified provider catalog — single source of truth for list/resolve APIs (Rust + Python)
• 11+ providers with dynamic, hybrid, or static discovery strategies
• Per-provider caching with configurable TTL and stale-on-error fallback
• Capability filtering — vision, tools, thinking, min/max input & output length, cost, deprecation
• Name search — exact ID, display name, provider/model, substring, and fuzzy matching
• Zero heuristics — capabilities from API responses or cited documentation only
• Static registry — 30+ models with verified data for offline/fallback use

See docs/discovery.md for full documentation.

Development

Local validation:

cargo fmt --all -- --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test --workspace --locked
cargo doc --workspace --no-deps --all-features

Python package validation:

cd edgequake-litellm
pip install . -v
pytest -q -k "not e2e"

Documentation

Specifications

Design specifications are tracked in specs/001-edgequake-llm/:

Upgrade specs for v0.9.0 are in specs/002-upgrade-support/, including the Provider Catalog API design.

Release

Release guides:

• docs/releasing.md: release checklist, tags, registry setup
• docs/release-cycle.md: end-to-end CI/CD flow
• CHANGELOG.md: release notes for the Rust crate
• edgequake-litellm/CHANGELOG.md: release notes for the Python package

Tag conventions:

• Rust crate: vX.Y.Z → crates.io
• Python package: py-vX.Y.Z → PyPI

Both publish workflows validate versions before publishing and attach release artifacts to GitHub Releases.

License

Apache-2.0. See LICENSE-APACHE.