dynamo

When to use Dynamo

• You're serving LLMs across multiple GPUs or nodes and need to coordinate them
• You want KV-aware routing to avoid redundant prefill computation
• You need to independently scale prefill and decode (disaggregated serving)
• You want automatic scaling that meets latency SLAs at minimum total cost of ownership (TCO)
• You need fast cold-starts when spinning up new replicas

If you're running a single model on a single GPU, your inference engine alone is probably sufficient.

Feature support at a glance:

Full Feature Matrix → — LoRA, request migration, speculative decoding, and feature interactions.

Key Results

What Dynamo Does

Most inference engines optimize a single GPU or a single node. Dynamo is the orchestration layer above them — it turns a cluster of GPUs into a coordinated inference system.

Architecture Deep Dive →

Core Capabilities

New in 1.0

• Zero-config deploy (DGDR) (beta): Specify model, HW, and SLA in one YAML — AISimulate performance models estimate candidate configurations, Planner optimizes the topology, and Dynamo deploys
• Agentic inference: Per-request hints for priority, expected output length, and speculative prefill, plus session metadata for tracing and SGLang subagent KV isolation. LangChain + NeMo Agent Toolkit integrations
• Multimodal E/P/D: Disaggregated encode/prefill/decode with embedding cache — 30% faster TTFT on image workloads
• Video generation: Native FastVideo + SGLang Diffusion support — real-time 1080p on single B200
• K8s Inference Gateway plugin: KV-aware routing inside the standard Kubernetes gateway
• Storage-tier KV offload: S3/Azure blob support + global KV events for cluster-wide cache visibility

Request Routing Topologies

Dynamo can expose traffic through two Kubernetes request routing topologies. Both expose an OpenAI-compatible API and support the same backends, disaggregated serving, and KV-aware routing.

Request flow for the Dynamo-native path is client → Frontend → Router → workers. Request flow for the Gateway API path is client → Gateway → EPP → Frontend sidecar (direct) → workers.

See the Gateway API Inference Extension (GAIE) guide for the Gateway API setup, supported features, and configuration.

Quick Start

This repo ships agent skills: if you work with an AI coding agent (Claude Code, Codex, Cursor), clone the repo and ask it to deploy, troubleshoot, benchmark, or optimize a Dynamo deployment. The skills activate automatically; no setup required.

Option A: Container (fastest)

Choose either vLLM or SGLang:

vLLM

# Pull a prebuilt container
docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.6.0

# Inside the container — start frontend and worker
python3 -m dynamo.frontend --http-port 8000 --discovery-backend file > /dev/null 2>&1 &
python3 -m dynamo.vllm --model Qwen/Qwen3-0.6B --discovery-backend file &

# Once the worker is ready, send a request
curl -s localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-0.6B",
  "messages": [{"role": "user", "content": "Hello!"}],
  "max_tokens": 100
}' | jq

SGLang

# Pull a prebuilt container
docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.6.0

# Inside the container — start frontend and worker
python3 -m dynamo.frontend --http-port 8000 --discovery-backend file > /dev/null 2>&1 &
python3 -m dynamo.sglang --model-path Qwen/Qwen3-0.6B --discovery-backend file &

# Once the worker is ready, send a request
curl -s localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-0.6B",
  "messages": [{"role": "user", "content": "Hello!"}],
  "max_tokens": 100
}' | jq

Also available: tensorrtllm-runtime:1.6.0.

Option B: Install from PyPI

Install uv (curl -LsSf https://astral.sh/uv/install.sh | sh), then choose your backend:

vLLM

uv pip install --prerelease=allow "ai-dynamo[vllm]"

SGLang

uv pip install --prerelease=allow "ai-dynamo[sglang]"

Note: TensorRT-LLM requires pip with --extra-index-url https://pypi.nvidia.com. See the install guide for TRT-LLM-specific instructions.

Then start the frontend and the matching worker as shown above. See the full installation guide for system dependencies and backend-specific notes.

Option C: Kubernetes (recommended)

For production multi-node clusters, install the Dynamo Platform and deploy with a single manifest:

# Zero-config deploy: specify model + SLA, Dynamo handles the rest
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: my-model
spec:
  model: Qwen/Qwen3-0.6B
  backend: vllm
  sla:
    ttft: 200.0   # ms
    itl: 20.0     # ms
  autoApply: true

Pre-built recipes for common models:

See recipes/ for the full list. Cloud-specific guides: AWS EKS · Google GKE · Azure AKS · Amazon ECS

Building from Source

For contributors who want to build and develop locally. See the full build guide for details.

# Install system deps (Ubuntu 24.04)
sudo apt install -y build-essential libhwloc-dev libudev-dev pkg-config libclang-dev protobuf-compiler python3-dev cmake

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh && source $HOME/.cargo/env

# Create venv and build
uv venv dynamo && source dynamo/bin/activate
uv pip install pip 'maturin[patchelf]'
cd lib/bindings/python && maturin develop --uv && cd $PROJECT_ROOT
uv pip install -e lib/gpu_memory_service
uv pip install -e .

VSCode/Cursor users: see the .devcontainer for a pre-configured dev environment.

Community and Contributing

Dynamo is built in the open with an OSS-first development model. We welcome contributions of all kinds.

• Contribution Guide — How to contribute code, docs, and recipes
• Design Proposals — RFCs for major features, tracked as dep:* labeled GitHub issues
• Office Hours — Biweekly calls
• Community Meetings (Youtube) – Weekly (Tue 9:00 AM PT) development community meetings
• Slack — Chat with the team and community
• Dynamo Day Recordings — Deep dives from production users

Latest News

• [03/15] Dynamo 1.0 is here — production-ready with strong community adoption
• [03/15] NVIDIA Blackwell Ultra sets new inference records in MLPerf
• [03/15] NVIDIA Blackwell leads on SemiAnalysis InferenceMax benchmarks
• [12/05] Moonshot AI's Kimi K2 achieves 10x inference speedup with Dynamo on GB200
• [12/02] Mistral AI runs Mistral Large 3 with 10x faster inference using Dynamo
• [11/20] Dell integrates PowerScale with NIXL for 19x faster TTFT

Older news

Dynamo provides comprehensive benchmarking tools:

• Benchmarking Guide – Compare deployment topologies using AIPerf
• SLA-Driven Deployments – Optimize deployments to meet SLA requirements

Frontend OpenAPI Specification

The OpenAI-compatible frontend exposes an OpenAPI 3 spec at /openapi.json. To generate without running the server:

cargo run -p dynamo-llm --bin generate-frontend-openapi

This writes to docs/reference/api/openapi.json.

Service Discovery and Messaging

Dynamo uses TCP for inter-component communication. On Kubernetes, native resources (CRDs + EndpointSlices) handle service discovery. External services are optional for most deployments:

Note: KV-aware routing does not require NATS. Enable KV events when you need event-backed cache-state tracking, or use --no-router-kv-events for prediction-based routing without external event infrastructure.

For Slurm or other distributed deployments that choose etcd or NATS JetStream-backed modes:

• etcd can be run directly as ./etcd.
• nats needs JetStream enabled: nats-server -js.

To quickly setup both: docker compose -f dev/docker-compose.yml up -d

More News

• [11/20] Dell integrates PowerScale with Dynamo's NIXL for 19x faster TTFT
• [11/20] WEKA partners with NVIDIA on KV cache storage for Dynamo
• [11/13] Dynamo Office Hours Playlist
• [10/16] How Baseten achieved 2x faster inference with NVIDIA Dynamo
• [12/01] InfoQ: NVIDIA Dynamo simplifies Kubernetes deployment for LLM inference

Reference

• Support Matrix — Hardware, OS, CUDA, and backend versions
• Feature Matrix — Detailed backend compatibility
• Release Artifacts — Containers, wheels, Helm charts
• Service Discovery — K8s-native vs etcd vs file-based discovery
• Benchmarking Guide — Compare deployment topologies with AIPerf