litellm-rs

Or install the gateway binary:

cargo install litellm-rs --version 0.10.1 --locked --bin gateway --no-default-features \
  --features postgres,sqlite,redis,s3,metrics,tracing,websockets,providers-extra,providers-extended,mcp-validation,a2a
mkdir -p config
curl -L https://raw.githubusercontent.com/majiayu000/litellm-rs/v0.10.1/config/gateway.dev.yaml.example -o config/gateway.yaml
gateway

The development config starts without provider credentials or auth secrets and uses the local vllm catalog provider. Use config/gateway.yaml.example for production-style deployments with real provider keys and auth enabled. The explicit sqlite feature enables the storage needed by the gateway binary. The registry install selects the same feature profile as release archives and containers, including MCP, A2A and Realtime. Binary and example config versions must match. See the 0.9 compatibility and rollout notes and versioned release assets.

The gateway serves its stable inference contract at GET /openapi.json; the versioned source is docs/openapi/inference.json. The admin control-plane contract is served at GET /admin/openapi.json (admin-authenticated) from docs/openapi/admin.json.

API-key lifetime budgets use an authoritative SQL reservation ledger. Create a key with max_budget, then inspect its cap and spent-plus-reserved amount at GET /v1/keys/{id}/usage. Gateways sharing the same database share one balance; restarts preserve both settled spend and outstanding holds. Redis provider/model budgets remain a separate scope. See the budget capability boundary for persistence requirements and unknown-usage behavior.

For a first HTTP request or setup failure, use the gateway setup and compatibility questions.

Supported Product Surfaces

Container distribution status

Install version 0.10.1 from the versioned crate, GitHub Release archives and ghcr.io/majiayu000/litellm-rs:0.10.1 (Linux amd64/arm64). These distributions use the shipped gateway feature profile documented below. Prefer a versioned image tag or immutable digest; latest is mutable. Docker Hub publication is optional. The 0.8 verification record is historical evidence for 0.8 artifacts, not verification of later releases.

Rust Crate

[dependencies]
litellm-rs = { version = "0.10", default-features = false, features = ["lite"] }

No make step is required for crate consumers.

Library Example

Add Tokio with macros and rt-multi-thread features. The complete hello example constructs a local vLLM deployment and installs its RuntimeBinding before calling completion. Run it against an existing local OpenAI-compatible server, passing its endpoint and served model:

cargo run --example hello -- http://localhost:8000/v1 your-served-model

tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

Copy examples/hello.rs into your application's src/main.rs. It uses public crate APIs and no provider credentials. The explicit private_network access policy permits the configured local endpoint; configure credentials and endpoint policy separately when integrating a hosted provider.

For a credential-free local HTTP check of the same example source as an independent crate consumer, run python3 scripts/test/library_consumer_smoke.py. The smoke uses this checkout as a path dependency; it does not verify a published registry artifact or a real provider account.

Gateway Configuration

The gateway router config maps these fields into the runtime router:

• router.strategy selects the deployment routing strategy.
• router.circuit_breaker.failure_threshold controls consecutive failures before cooldown.
• router.circuit_breaker.recovery_timeout controls cooldown duration in seconds.
• router.circuit_breaker.min_requests sets the sample size required before cooldown.
• router.circuit_breaker.success_threshold sets the successes required to recover from cooldown.
• router.load_balancer.health_check_enabled enables pre-call deployment health checks.

Active provider probes are opt-in: a provider's health_check must differ from the defaults. Native probes are limited to provider types whose configured deployment is unambiguously a chat model (currently Anthropic and GitHub Copilot); they send a one-token request and may incur provider charges. OpenAI, Bedrock, OpenAI-compatible, Vertex AI, FalAI, and other multi-capability providers require a custom health_check.endpoint. Without an active probe policy, readiness remains fail-closed (Unknown).

router.load_balancer.sticky_sessions and router.load_balancer.session_timeout are reserved for future session affinity. Non-default values fail config validation until runtime affinity is implemented.

Gateway YAML can publish stable model names and deterministic primary/fallback tiers:

providers:
  - name: openai-primary
    provider_type: openai
    api_key: "${OPENAI_API_KEY}"
    models: [gpt-4o]
    priority: 0
  - name: openai-fallback
    provider_type: openai
    api_key: "${OPENAI_API_KEY}"
    models: [gpt-4o]
    priority: 10

model_aliases:
  production-chat: gpt-4o
  stable-chat: production-chat

router:
  strategy: priority_based

Alias chains are validated and flattened at startup; empty values, cycles, canonical-name collisions, and targets without an enabled deployment fail startup. Alias names appear in /v1/models alongside canonical models. Lower numeric priority wins under priority_based; omitted provider priorities default to 0. When rolling back to a binary that predates these fields, remove model_aliases and priority from YAML before rolling back the binary, because unknown fields are rejected.

Core Subsystem Runtime Status

Runtime wiring decisions are tracked in src/core/subsystem_registry.rs, and tests assert that every module exported from src/core/mod.rs is either referenced by the gateway runtime or explicitly classified. The current issue-838 subsystem decisions are:

Installation

Since 0.8.0, default features select the reusable library with metrics and tracing. Gateway and database dependencies are opt-in. The previous 0.7.0 release used gateway defaults; gateway consumers must explicitly select a storage feature such as sqlite or postgres.

# Library with metrics and tracing
[dependencies]
litellm-rs = { version = "0.10", default-features = false, features = ["lite"] }

# Library without optional observability
[dependencies]
litellm-rs = { version = "0.10", default-features = false }

# Gateway with SQLite (also enables Redis support)
[dependencies]
litellm-rs = { version = "0.10", features = ["sqlite"] }

Use the crate root or litellm_rs::prelude for completion and embedding APIs. Advanced types have explicit module paths: core::providers for provider construction, core::router for deployment management, and core::models::openai for HTTP wire models. Hidden root re-exports of these advanced types have been removed in 0.8.0.

Provider Support

Perplexity's old Sonar Chat Completions endpoint retired on 2026-09-27. The perplexity / perplexity-ai / pplx selectors are not currently dispatchable; the replacement Agent API requires a native protocol adapter. Historical prices remain available and do not establish callable models. See the retirement audit.

GitHub Models (github / github-models) retired on 2026-07-30 and is no longer a callable selector. GitHub Copilot remains a separate native provider. See the retirement audit.

Lambda shared inference retired on 2025-09-25. The lambda_ai selector is removed. The unverified meta_llama, v0 and amazon_nova default transports are also disabled; use the existing Bedrock provider for supported Nova models or configure an OpenAI-compatible deployment with its real endpoint. See the remaining catalog review.

Providers are organised into two tiers (see CLAUDE.md → Provider Tiers for the engineering definition).

• Tier 1 — catalog-only: OpenAI-compatible endpoints declared as data in src/core/providers/registry/catalog.rs. Routed through OpenAILikeProvider. Always available (no cargo feature required). The runtime supports chat and streaming plus explicitly verified embeddings, images and audio capabilities for selected providers; see compatible non-chat support. Capability declarations do not imply that every model supports every endpoint.
• Tier 2 — code-based: providers with custom request/response handling, auth signing, or streaming. Wired into the Provider enum and the factory. Some Tier 2 builders are feature-gated.

External Rust providers implement ExternalProvider and register through Provider::External(Arc::new(provider)), Deployment::new, and UnifiedRouter::add_deployment, without editing the internal enum. See the external provider integration tests for capability, health, error and streaming behavior. This is an in-process API; OpenAI-compatible servers can use the existing configuration path.

The provider and legacy adapter matrices below are validated against the provider registry and Tier 1 catalog. The source of truth for Tier 1 entries is catalog.rs; Tier 2 identity and dispatch metadata lives in src/core/providers/registry/types.rs, with construction branches in src/core/providers/factory/registry.rs. Legacy adapter availability lives in src/core/providers/registry/support_matrix.rs. passthrough means a retained adapter forwards the call to the upstream OpenAI-compatible endpoint without per-provider transformation.

Legacy adapter matrix

This matrix records selector-based compatibility adapters. It is not the capability source for LLMClient::from_runtime, DefaultRouter::from_runtime, or the free completion() functions. Canonical runtime support is derived from the selected deployment's ProviderCapability; unsupported capabilities fail closed with a typed provider error.

The runway-media feature exposes a library-only Runway task contract for text/image-to-video submit, query, cancel, and result polling. It does not mount a gateway video route.

Tier 2 — code-based providers

GitHub Copilot captures its effective token directory, credential filenames, and file contents when the provider is constructed. Reload the gateway after changing those paths or switching credential files to another account. Existing requests and provider clones keep their captured credentials; API-key refresh uses that resource's access token and returns a matching token and endpoint. Authentication remains lazy when no credentials are available. If only an API-key cache was captured, configure an access token and reload before that key needs refreshing.

Tier 1 — catalog providers (OpenAI-compatible, always available)

All entries below route through OpenAILikeProvider. Chat and streaming use the compatible /chat/completions protocol. Selected providers also expose verified non-chat capabilities listed in compatible non-chat support; other capabilities fail explicitly.

Cloud (Bearer auth via env var):

groq, ai21, huggingface, baseten, together, together_ai, fireworks, fireworks_ai, cerebras, openrouter, deepinfra, deepseek, novita, nvidia_nim, nebius, nscale, hyperbolic, featherless, galadriel, sambanova, heroku, friendliai, xai, moonshot, dashscope, qwen, baichuan, minimax, volcengine, xiaomi_mimo, zhipu, zai, lemonade, linkup, poe, wandb, nanogpt, aiml_api, aiml, aleph_alpha, anyscale, bytez, comet_api, compactifai, maritalk, siliconflow, yi, ovhcloud

Local (no API key):

vllm, hosted_vllm, lm_studio, llamafile, docker_model_runner, xinference, infinity, oobabooga

For self-hosted or unlisted OpenAI-compatible endpoints, prefer the generic openai_compatible provider type instead.

Environment Variables

# Provider API Keys
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_API_KEY=...
AZURE_OPENAI_API_KEY=...
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
AWS_REGION=us-east-1
GROQ_API_KEY=...
AI21_API_KEY=...
HF_TOKEN=...
BASETEN_API_KEY=...
DEEPSEEK_API_KEY=...
MOONSHOT_API_KEY=...
ZHIPU_API_KEY=...
MINIMAX_API_KEY=...

# Optional
LITELLM_VERBOSE=true  # Enable verbose logging

Examples

Multi-Provider Routing

use litellm_rs::{completion, user_message};

// Automatically routes to the right provider based on model name
let openai = completion("gpt-5.5", vec![user_message("Hi")], None).await?;
let anthropic = completion("anthropic/claude-opus-4-8", vec![user_message("Hi")], None).await?;
let groq = completion("groq/llama-3.1-8b-instant", vec![user_message("Hi")], None).await?;
let bedrock = completion(
    "bedrock/us.anthropic.claude-3-5-sonnet-20241022-v2:0",
    vec![user_message("Hi")],
    None,
)
.await?;

bedrock/ uses the native AWS Bedrock provider. It signs requests with AWS SigV4 and preserves AWS execution model IDs such as us.*, global.*, region-prefixed IDs, and Bedrock ARNs. Use openai_compatible for Bedrock Access Gateway or other OpenAI-compatible proxies instead.

Embeddings

use litellm_rs::{embedding, embed_text};

// Single text
let embedding = embed_text("text-embedding-3-small", "Hello world").await?;

// Batch
let embeddings = embedding(
    "text-embedding-3-small",
    vec!["Hello", "World"],
    None,
).await?;

Streaming

use litellm_rs::{completion_stream, user_message};
use futures::StreamExt;

let mut stream = completion_stream(
    "gpt-4",
    vec![user_message("Tell me a story")],
    None,
).await?;

while let Some(chunk) = stream.next().await {
    if let Ok(chunk) = chunk {
        print!("{}", chunk.choices[0].delta.content.unwrap_or_default());
    }
}

Performance

Gateway throughput and latency claims require a dated raw artifact tied to an exact Git revision. See the reproducible gateway-overhead benchmark methodology for the fixed workload, commands, environment metadata, percentile reporting, and artifact format.

The Criterion benchmarks under benches/ measure in-process components; they must not be presented as end-to-end HTTP gateway throughput.

Troubleshooting

Build/test uses too much CPU or memory

• Use API-only defaults first: cargo test --lib --tests --no-default-features --features "lite"
• Limit local parallelism when needed: CARGO_BUILD_JOBS=4 cargo test --lib --tests --no-default-features --features "lite" -- --test-threads=4
• Avoid --all-features unless you are doing release/nightly validation

I only need provider API aggregation, not gateway

• Prefer default-features = false with features = ["lite"]
• Use gateway runtime commands only when you need HTTP server/auth/storage middleware

Documentation

• API Documentation
• Documentation Index
• Development Gateway Config
• Production Gateway Config
• Examples

Contributing

See CONTRIBUTING.md for development setup and guidelines.

Security

See SECURITY.md for security policy and vulnerability reporting.

The Agent Infra Stack

This project is one layer of an open-source stack for running coding agents (Claude Code, Codex) as serious infrastructure. Every piece works standalone; together they close the loop:

litellm-rs is the Route layer — a self-hosted OpenAI-compatible gateway. See Provider Support for runtime and adapter coverage.

License

MIT License - see LICENSE for details.

Acknowledgments

Inspired by LiteLLM (Python).