smg
App Listed
High-performance model-routing gateway for large-scale LLM deployments
- Downloads
- 116
- Dependents
- Not applicable
- Stars
- 703
- Forks
- 212
- Contributors
- 166
- Commits
- 4,847
- Latest release
- 1.14.0 ·
About
Shepherd Model Gateway
Engine-agnostic, high-performance model-routing gateway for large-scale LLM deployments. SMG centralizes worker lifecycle management, balances traffic across self-hosted engines and cloud providers, and gives you enterprise-grade control over multi-tenancy, chat-history storage, MCP tooling, and observability — behind one unified endpoint.
Read the full README
Why SMG?
| 🚀 Maximize GPU Utilization | Cache-aware routing tracks each worker's KV-cache state in radix trees to reuse prefixes across SGLang, vLLM, TensorRT-LLM, TokenSpeed, and MLX — with load modeling that accounts for queued token work and KV pressure. |
| 🔌 One API, Any Backend | Route to self-hosted engines over HTTP or gRPC, or to OpenAI, Anthropic, Gemini, and xAI — plus any OpenAI-compatible endpoint — through a single unified gateway. |
| ⚡ Built for Speed | Native Rust with streaming gRPC pipelines, cached tokenization with zero-copy cache hits, prefill/decode disaggregation (including a separate encode stage for vision), and DP-aware routing for data-parallel engines. |
| 🔒 Enterprise Control | Priority admission scheduling with preemption and per-tenant controls, API-key auth with OIDC on the control plane, WebAssembly plugins for custom logic, and chat history that never leaves your infrastructure. |
| 📊 Full Observability | 90+ Prometheus metrics, OpenTelemetry tracing with W3C trace context propagated into the engines over both HTTP and gRPC, and structured JSON logs with request correlation. |
API Coverage: OpenAI Chat Completions, Completions, Embeddings, Rerank, and Classify; Responses and Conversations APIs for agents; Anthropic Messages; Gemini Interactions; Realtime over WebSocket and WebRTC; audio transcription; tokenize/detokenize; and MCP tool execution with approval policies in the Responses and Messages APIs.
Quick Start
Install — pick your preferred method:
# Docker
docker pull lightseekorg/smg:latest
# Kubernetes (Helm)
helm install smg oci://ghcr.io/smg-project/charts/smg
# Python
pip install smg
# Rust (needs protoc)
cargo install smg
Run — point SMG at your inference workers:
# Single worker
smg launch --worker-urls http://localhost:8000
# Multiple workers with cache-aware routing
smg launch --worker-urls http://gpu1:8000 http://gpu2:8000 --policy cache_aware
# With high availability mesh
smg launch --worker-urls http://gpu1:8000 --enable-mesh \
--mesh-advertise-host 10.0.0.1 --mesh-peer-urls 10.0.0.2:39527
Use — send requests to the gateway:
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "llama3", "messages": [{"role": "user", "content": "Hello!"}]}'
That's it. SMG is now load-balancing requests across your workers.
Supported Backends
| Self-Hosted Engines | vLLM · SGLang · TokenSpeed · TensorRT-LLM · MLX (Apple Silicon) · any OpenAI-compatible server (e.g. Ollama) |
| Cloud Providers | OpenAI · Anthropic · Google Gemini · xAI · OCI Generative AI · AWS Bedrock · Azure OpenAI · any OpenAI-compatible provider (Groq, Together, …) |
Features
| Feature | Description |
|---|---|
| 10 Routing Policies | cache_aware, least_load, power_of_two, consistent_hashing, prefix_hash, bucket, round_robin, random, manual, passthrough |
| gRPC Pipeline | Native streaming gRPC to the engines with prefill/decode and encode disaggregation and DP-aware routing |
| Kubernetes Discovery | Native pod watchers with label selectors, per-role prefill/decode/encode selectors, and router peer discovery |
| Model Parsers | 21 tool-call parsers and 16 reasoning parsers with automatic model detection — DeepSeek, Qwen, Kimi, GLM, Llama, Mistral, Command, Nemotron, and more |
| MCP Integration | Tool discovery and execution over stdio, SSE, and streamable HTTP, with approval policies and audit logging |
| High Availability | Mesh networking with SWIM gossip and CRDT-replicated state for multi-node deployments |
| Chat History | Pluggable storage with schema migrations: PostgreSQL, Oracle, Redis, or in-memory |
| WASM Plugins | Extend request and response handling with custom WebAssembly middleware |
| Resilience | Circuit breakers, retries with backoff and jitter, rate limiting, and priority admission scheduling |
Documentation
Full documentation lives at lightseek.org/smg.
| Getting Started | Installation and first steps |
| Architecture | How SMG works |
| Configuration | CLI reference and options |
| API Reference | OpenAI-compatible endpoints |
| Kubernetes Setup | In-cluster discovery and production setup |
Contributing
We welcome contributions! See the Contributing Guide for details.
Compatibility
- Runs on
- GPU support unknown
What shows it
- no dependency or feature matches a backend row
- Native code
- Required
- Binds native code: k8s-openapi, odpic-sys, openssl-sys, tikv-jemalloc-sys, nixl-sys (feature nixl)
- Build uses a native toolchain: cc, pkg-config
- Python bindings
- No Python binding crate found
- License
- Apache-2.0
Compare with similar projects
Order: strongest tier first, then most downloads, then name, with unmeasured values last. A library counts downloads over 30 days. An app counts binary downloads per release. An app with none counts its crate downloads over 30 days. ML Rust runs this catalogue, so its closest project always gets a row.
| Project | Downloads | Dependents | Stars | Latest release | Runs on (GPU) | Native code | License |
|---|---|---|---|---|---|---|---|
|
High-performance model-routing gateway for large-scale LLM deployments |
116 downloads over 30 days | Not applicable | 703 | 1.14.0 | GPU support unknown | Binds native code: k8s-openapi, odpic-sys, openssl-sys, tikv-jemalloc-sys, nixl-sys (feature nixl). Build uses a native toolchain: cc, pkg-config | Apache-2.0 |
|
Terminal UI to chat with large language models (LLM) using different model backends, and integrations with your favourite editors! |
5,642 binary downloads per release | Not applicable | 772 | 0.13.0 | GPU support unknown | No native code found | MIT |
|
Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic. |
211 binary downloads per release | Not applicable | 7,111 | Not measured | Not measured | Apache-2.0 | |
|
llm_client: The easiest Rust interface for local LLMs |
43 downloads over 30 days | 1 | 254 | 0.0.7 | Metal | Binds native code: nvml-wrapper-sys | MIT |
|
A high-performance AI Gateway written in Rust, providing OpenAI-compatible APIs with intelligent routing, load balancing, and enterprise features |
34 binary downloads per release | Not applicable | 123 | 0.10.1 | GPU support unknown | No native code found | MIT |
|
icebreaker Listed
A local AI chat app powered by 🦀Rust, 🧊iced, 🤗Hugging Face, and 🦙llama.cpp |
12 downloads over 30 days | 0 | 468 | 2026.2.0 | GPU support unknown | No native code found | MIT |
|
Inference server for LLMs, speech and embeddings: OpenAI-compatible routes with streaming, plus speech synthesis (TTS) and transcription (ASR). Serves Llama, Mistral, Qwen, Mamba2, MLA+MoE and hybrid models from SafeTensors, AWQ, GPTQ and GGUF. |
8 binary downloads per release | Not applicable | 7 | 0.7.1 | CUDA, WebGPU | Binds native code: ring (feature distributed) | Apache-2.0 |
All projects in Build LLM apps and agentsAll projects in Run and serve models
Why ListedNo archive, New or Featured rule places it, so it is Listed.
What it lacks for Featured
- The check holds no value for Homebrew installs on request over 365 days, Repology repository families, Docker pull growth over 90 days. A gate that lacks its signal does not pass.
What else the check found
- The last activity is on 9 Nov 2026, 0 days ago. The rules expect an update within 365 days. Dormancy rule
Not measured: Homebrew installs on request over 365 days, Repology repository families, Docker pull growth over 90 days.
Exact rules: Listed, Featured app. The check on applied thresholds version 9.
Every rule outcome, as the check records it
expected-interval: app: E = 365 daysdormancy: last activity 2026-11-09, 0 days ago, E = 365: activefeatured: not featured: unknown homebrew_on_request_365d, repology_families, docker_pull_growth_90dlisted: listed: no archive, new or featured rule applies
Crates24 crates
- smg Primary
- data-connector
- engine-servicer
- engine-zmq-adapter
- engine-zmq-client
- kv-index
- llm-multimodal
- llm-tokenizer
- openai-protocol
- reasoning-parser
- smg-auth
- smg-client
- smg-external-router
- smg-grpc-client
- smg-mcp
- smg-mesh
- smg-mm-rdma
- smg-radix-tree
- smg-response-template
- smg-rl
- smg-symphony
- smg-wasm
- tool-parser
- wfaas
Measurements and sources
Each signal row shows the raw value, its source and the fetch date. The Sources list gathers every distinct source on this page. Movement needs more history.
Sources6 sources
- Description: the crates.io description
- crates.io database dump, fetched
- GitHub releases, fetched
- GitHub repository, fetched
- crates.io package, fetched
- GitHub contributors, fetched
Adoption0 binary downloads per release4 not measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Active dependents | Not measured | Yes | ||
| New dependents (365 days) | Not measured | Yes | ||
| Peak active dependents | Not measured | Yes | ||
| Active dependents as a share of the peak | Not measured | Yes | ||
| Binary downloads per release | 0 | GitHub releases | Yes |
MaintenanceLatest human commit: on the check date4 not measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Latest human commit | 2026-11-09 | GitHub repository | Yes | |
| Latest release | 2026-11-03 | GitHub releases | Yes | |
| First release | 2026-01-06 | crates.io database dump | Yes | |
| Latest version | 1.14.0 | crates.io database dump | No | |
| Days the oldest outside issue waits for a reply | 25 | GitHub repository | Yes | |
| Maintainer replies over 365 days | 73 | GitHub repository | Yes | |
| Repository archived | false | GitHub repository | Yes | |
| Repository gone | false | GitHub repository | Yes | |
| Release commit or manifest in the repo | true | crates.io package | No | |
| Published docs | Not measured | Yes | ||
| Latest release fails cargo check | Not measured | Yes | ||
| Build check date | Not measured | Yes | ||
| Build check toolchain | Not measured | Yes |
SecurityNot measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Unpatched vulnerabilities past the grace window | Not measured | Yes | ||
| RustSec unmaintained advisory | Not measured | Yes | ||
| Deprecation notice | Not measured | Yes | ||
| Deprecated requirement | Not measured | Yes |
Distribution23 releases, 116 downloads over 30 days6 not measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Releases | 23 | crates.io database dump | No | |
| Downloads over 30 days | 116 | crates.io database dump | No | |
| Binary downloads over the last 3 releases | 0 | GitHub releases | Yes | |
| Homebrew installs on request over 365 days | Not measured | Yes | ||
| Repology repository families | Not measured | Yes | ||
| Docker pull growth over 90 days | Not measured | Yes | ||
| Docker pulls at the growth baseline | Not measured | Yes | ||
| Every version a prerelease | false | crates.io database dump | Yes | |
| Every version 0.0.x | false | crates.io database dump | Yes | |
| Every version yanked | Not measured | Yes | ||
| Crate deleted from crates.io | Not measured | Yes |
Repository703 stars, 212 forks, 166 contributors, 4,847 commits. Stars alone never qualify a project.
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| GitHub stars | 703 | GitHub repository | Yes — one part of the repository route | |
| GitHub forks | 212 | GitHub repository | Yes — one part of the repository route | |
| Contributors with a GitHub account | 166 | GitHub contributors | Yes — one part of the repository route | |
| Commits on the default branch | 4847 | GitHub repository | No | |
| Repository's main crate left out of the entry | none | crates.io database dump | Yes |
Facts that never count
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| License | Apache-2.0 | crates.io database dump | No | |
| Runs on, from listed crates and their direct dependencies | GPU support unknown | crates.io database dump | No | |
| What shows each backend | no dependency or feature matches a backend row | crates.io database dump | No | |
| Native code, from listed crates and their dependencies | Binds native code: k8s-openapi, odpic-sys, openssl-sys, tikv-jemalloc-sys, nixl-sys (feature nixl). Build uses a native toolchain: cc, pkg-config | crates.io database dump | No | |
| Python bindings, from direct dependencies of listed crates | No Python binding crate found | crates.io database dump | No | |
| Crates that depend on a listed crate directly | Not measured | No |
Corrections and review history
No correction applies to this entry.
Judgments
A judgment enters only through an approved issue and a merged pull request.
No judgment applies to this entry.