candle-vllm

git clone git@github.com:EricLBuehler/candle-vllm.git
cd candle-vllm

# CUDA (11+, 12+, 13.0) — remove flashinfer,cutlass for sm_70/sm_75
cargo install --features cuda,nccl,flashinfer,cutlass --path .

# macOS/Metal
cargo install --features metal --path .

Option 3 — Docker

# Pass custom SM version and CUDA version: ./build_docker.sh "cuda,nccl,flashinfer,cutlass" sm_90 13.0.0
./build_docker.sh "cuda,nccl,flashinfer,cutlass"

▶️ Run

Using HuggingFace Model ID:

candle-vllm --m Qwen/Qwen3.6-27B-FP8 --ui-server

candle-vllm --m unsloth/Qwen3.5-122B-A10B-GGUF --f Q3_K_S --d 0,1 --ui-server

candle-vllm --m zai-org/GLM-5.2-FP8 --d 0,1,2,3,4,5,6,7 --ui-server

Using local model path:

# Local safetensors directory
candle-vllm --d 0,1,2,3,4,5,6,7 --m /home/data/GLM-5.2-FP8/ --ui-server

# Local GGUF file (single or split-shard)
candle-vllm --d 0,1 --m /home/data/model-Q4_K_M.gguf --ui-server

# Local directory containing GGUF files (auto-detected)
candle-vllm --d 0,1 --m /home/data/Qwen3.5-35B-A3B-GGUF/ --ui-server

Tip: Add --ui-server to launch the built-in ChatGPT-style Web UI. The UI server uses the API port minus one (e.g., API on 2000, UI on 1999).

📈 Performance

Single-request decode speed (input 4k, output 1k, on Hopper 80G)

Demo Video — GPU & Apple Silicon

Chat demo on GPU (A100, BF16, QWen3-8B Reasoning Model)

Chat demo on Apple Silicon (M4, 16GB unified memory, Q2K, QWen3-8B)

🧠 Features

• OpenAI compatible API server for serving LLMs
• Streaming support in generation
• Efficient KV cache management with PagedAttention
• Continuous batching (batched decoding for incoming requests over time)
• In-situ quantization (and In-situ Marlin format conversion)
• GPTQ/Marlin format quantization (4-bit)
• Support Mac/Metal devices
• Support Multi-GPU inference (both multi-process and multi-threaded mode)
• Support Multi-node inference via TCP-based coordination
• Support Chunked Prefilling (default chunk size 8K)
• Support CUDA Graph
• Support Qwen3.5 MTP speculative decoding with CUDA Graph via --mtp
• Support external DFlash2 speculative decoding via --draft-model and --num-speculative-tokens
• Support Model Context Protocol (MCP) and OpenAI-compatible tool calling
• Support Prefix Caching
• Support Block-wise FP8 Models (SM90+, Qwen3 Series)
• Support FP8 KV Cache on all CUDA and Metal platforms
• Support TurboQuant KV Cache (turbo8/turbo4/turbo3) with native flash attention kernels
• Support Flashinfer Backend
• Support manual YaRN RoPE scaling override via --yarn-scaling-factor
• Support MXFP4/NVFP4 models
• Support GPTQ/AWQ/Marlin models, including pack-quantized AWQ MoE
• Support DeepSeek V3.2 and GLM-5.2 FP8 models

📘 Usage

Running Models

Tip: By default, candle-vllm starts an OpenAI-compatible API server at http://localhost:2000. Add --ui-server to also launch the built-in ChatGPT-style Web UI.

# FP8 model + Web UI
candle-vllm --m Qwen/Qwen3.8-27B-FP8 --ui-server

# GLM-5.2 FP8 model
candle-vllm --d 0,1,2,3,4,5,6,7 --m zai-org/GLM-5.2-FP8 --ui-server

# Unquantized Safetensors (multi-GPU)
candle-vllm --d 0,1 --w /home/Qwen3-30B-A3B-Instruct-2507/

# ISQ on-the-fly quantization
candle-vllm --m Qwen/Qwen3.6-27B --isq q4k

# FP4 Model
candle-vllm --m GadflyII/GLM-4.7-Flash-NVFP4 --ui-server

# GGUF model
candle-vllm --m unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF --f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --ui-server

# Manual YaRN scaling
candle-vllm --m Qwen/Qwen3.6-35B-A3B --yarn-scaling-factor 4.0 --ui-server

FP8 / FP4 models

# FP8 Model (block-wise quant, build with cutlass feature)
candle-vllm --m Qwen/Qwen3.8-27B-FP8 --ui-server

# Faster GDN prefill on Hopper with slight precision loss
SM90_LOWER_PRECISION_GDN_PREFILL=1 candle-vllm --m Qwen/Qwen3.5-35B-A3B-FP8

# Qwen3.5 MTP speculative decoding (2 draft tokens per step)
candle-vllm --w /data/Qwen3.5-35B-A3B-FP8/ --mtp 2 --ui-server

# External DFlash2 speculative decoding
candle-vllm --w /data/Qwen3.5-target/ --draft-model /data/Qwen3.5-DFlash2/ --num-speculative-tokens 7 --ui-server

# GLM-5.2 FP8 Model
candle-vllm --d 0,1,2,3,4,5,6,7 --m zai-org/GLM-5.2-FP8 --ui-server

# FP8 on MacOS/Metal (Dense)
candle-vllm --m Qwen/Qwen3-4B-Instruct-2507-FP8 --ui-server

# FP4 Model (MXFP4/NVFP4, MLX quantized format not supported)
candle-vllm --m GadflyII/GLM-4.7-Flash-NVFP4 --ui-server

# MXFP4
candle-vllm --m nm-testing/Qwen3-30B-A3B-MXFP4A16 --ui-server

GGUF models

# Local GGUF file via --m (recommended)
candle-vllm --m /home/data/Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --ui-server

# Local GGUF file via --f (legacy)
candle-vllm --f /home/data/Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --ui-server

# Local directory containing GGUF (auto-detected, mmproj loaded on demand)
candle-vllm --m /home/data/Qwen3.5-35B-A3B-GGUF/ --ui-server

# From HuggingFace (exact file)
candle-vllm --m unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF --f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --ui-server

# From HuggingFace (subfolder — downloads all GGUF files in the remote path)
candle-vllm --m unsloth/Qwen3.5-122B-A10B-GGUF --f Q3_K_S --d 0,1 --ui-server

# Local multi-shard GGUF (split files auto-discovered from local path)
candle-vllm --m /home/data/model-00001-of-00003.gguf --d 0,1 --ui-server

# GGUF on Apple Silicon
candle-vllm --m /home/qwq-32b-q4_k_m.gguf --ui-server
candle-vllm --m Qwen/QwQ-32B-GGUF --f qwq-32b-q4_k_m.gguf --ui-server

Multi-shard GGUF: Split GGUF files (e.g., model-00001-of-00005.gguf) are automatically discovered — both locally (from the same directory) and remotely (from the HuggingFace repo). When --f is a subfolder name (not ending in .gguf), all GGUF files in that remote subfolder are downloaded. Vision tower auxiliary files (mmproj*.gguf) are loaded on demand for multimodal models.

ISQ (In-situ quantization)

Simply add --isq parameter when running unquantized models:

candle-vllm --m Qwen/Qwen3.6-27B --isq q4k

Options: q4_0, q4_1, q5_0, q5_1, q8_0, q2k, q3k, q4k, q5k, q6k

GPTQ / AWQ / Marlin models

# Marlin-compatible GPTQ (4-bit, 128-group, desc_act=False)
candle-vllm --m thesven/Llama-3-8B-GPTQ-4bit

# Convert uncompressed model to Marlin-compatible format
python3 examples/convert_marlin.py --src /home/DeepSeek-R1-Distill-Qwen-14B/ --dst /home/DeepSeek-R1-Distill-Qwen-14B-GPTQ_4bit-128g
candle-vllm --w /home/DeepSeek-R1-Distill-Qwen-14B-GPTQ_4bit-128g

# Convert AWQ to Marlin-compatible format
python3 examples/convert_awq_marlin.py --src /home/Meta-Llama-3.1-8B-Instruct-AWQ-INT4/ --dst /home/Meta-Llama-3.1-8B-Instruct-AWQ-INT4-Marlin/ --bits 4 --method awq --group 128 --nk False
candle-vllm --d 0 --w /home/Meta-Llama-3.1-8B-Instruct-AWQ-INT4-Marlin/

# AWQ MoE (pack-quantized), e.g. Qwen3-Coder-Next
candle-vllm --m cyankiwi/Qwen3-Coder-Next-AWQ-4bit

# Direct Marlin-format model
candle-vllm --w /home/DeepSeek-R1-Distill-Qwen-14B-GPTQ-Marlin/

🗜️ TurboQuant KV Cache

TurboQuant compresses the KV cache using Walsh-Hadamard transform for higher throughput and longer context:

# Turbo4 (4-bit KV cache, ~3.7x compression)
candle-vllm --w /data/Qwen3.5-27B-FP8/ --kvcache-dtype turbo4

# Turbo8 (FP8 K + 4-bit V, ~2.6x compression)
candle-vllm --w /data/Qwen3.5-27B-FP8/ --kvcache-dtype turbo8

# Turbo3 (3-bit K + 4-bit V, ~4.7x compression)
candle-vllm --w /data/Qwen3.5-27B-FP8/ --kvcache-dtype turbo3

# FP8 KV Cache
candle-vllm --w /data/Qwen3.5-35B-A3B-FP8/ --kvcache-dtype fp8

Note: TurboQuant uses native flash attention kernels (flashinfer is automatically disabled). Supported on both CUDA (SM70+) and Metal (Apple Silicon) platforms. MLA models (DeepSeek, GLM4/GLM-5.2) auto-fallback to standard KV cache as TurboQuant is incompatible with their compressed KV layout.

🖥️ Multi-GPU Inference

Multi-process mode (recommended)

# QwQ-32B BF16 on two GPUs
candle-vllm --d 0,1 --w /home/QwQ-32B/

# QwQ-32B 4-bit AWQ on two GPUs
python3 examples/convert_awq_marlin.py --src /home/QwQ-32B-AWQ/ --dst /home/QwQ-32B-AWQ-Marlin/ --bits 4 --method awq --group 128 --nk False
candle-vllm --d 0,1 --w /home/QwQ-32B-AWQ-Marlin/

Note: Number of GPUs (--d) must be a power of 2 (e.g., 2, 4, or 8).

Multi-threaded mode (debug)

# Add --multithread parameter
candle-vllm --multithread --d 0,1 --w /home/QwQ-32B/

# Troubleshooting
export NCCL_P2P_DISABLE=1  # disable P2P if encountering illegal memory access

Speculative decoding: MTP and DFlash2

Candle-vLLM supports two speculative-decoding modes for Qwen3.5/3.6/3.8-family targets:

• Built-in MTP (Multi-Token Prediction): the target checkpoint contains mtp.* weights. Use --mtp N or the equivalent --num-speculative-tokens N.
• External DFlash2: use a matching DFlash2 draft checkpoint with --draft-model. If --draft-model is set, DFlash2 takes priority over MTP; --num-speculative-tokens optionally overrides the draft width (otherwise block_size - 1 from the draft config is used).

MTP example:

# Three MTP draft tokens per decode step
candle-vllm --m /data/Qwen3.5-35B-A3B-FP8/ \
  --num-speculative-tokens 3 --ui-server

DFlash2 example:

# Qwen3.8 target and its matching DFlash2 draft checkpoint
candle-vllm --m /data/Qwen3.8-27B-FP8/ \
  --draft-model /data/Qwen3.8-27B-DFlash2-FP8/ \
  --num-speculative-tokens 7 --ui-server

🌐 Multi-Node Inference

Distribute inference across multiple machines using TCP-based NCCL bootstrap. No MPI required.

# On master node (192.168.1.100):
candle-vllm --d 0,1,2,3,4,5,6,7 --w /data/DeepSeek-R1-AWQ-Marlin/ \
  --num-nodes 2 --node-rank 0 --master-addr 192.168.1.100 --master-port 29500

# On worker node (192.168.1.101):
candle-vllm --d 0,1,2,3,4,5,6,7 --w /data/DeepSeek-R1-AWQ-Marlin/ \
  --num-nodes 2 --node-rank 1 --master-addr 192.168.1.100 --master-port 29500

All nodes must have model weights locally and be TCP-reachable on --master-port (default 29500).

📐 NUMA Binding

Show command

sudo apt-get install numactl

# 8 GPUs, 2 NUMA nodes
MAP_NUMA_NODE=0,0,0,0,1,1,1,1 numactl --cpunodebind=0 --membind=0 candle-vllm --d 0,1,2,3,4,5,6,7 --w /home/data/DeepSeek-V2-Chat-AWQ-Marlin

# 4 GPUs
MAP_NUMA_NODE=0,0,0,0 numactl --cpunodebind=0 --membind=0 candle-vllm --d 0,1,2,3 --w /home/data/DeepSeek-V2-Chat-AWQ-Marlin

numactl --cpunodebind=0 --membind=0 specifies the master rank's NUMA binding and must match MAP_NUMA_NODE.

⚙️ CLI Reference

Binding examples:

candle-vllm --h 127.0.0.1 --p 8000 --m Qwen/Qwen3.6-27B-FP8
candle-vllm --h 127.0.0.1:8000 --m Qwen/Qwen3.6-27B-FP8
candle-vllm --h '[::1]:8000' --m Qwen/Qwen3.6-27B-FP8
candle-vllm --h unix:///tmp/candle-vllm.sock --m Qwen/Qwen3.6-27B-FP8

📚 Documentation

Using Agents under Candle-vLLM backend: xbot · OpenCode · Kilo Code

🛠️ Roadmap

• [x] OpenAI-compatible API server (streaming)
• [x] Continuous batching
• [x] Flash Attention (CUDA)
• [x] FlashInfer backend
• [x] CUDA Graph
• [x] Chunked Prefill
• [x] Prefix Caching (CUDA & Metal)
• [x] Multi-GPU inference (multi-process & multi-threaded)
• [x] Multi-node tensor parallelism (TCP-based NCCL, no MPI)
• [x] In-situ quantization (GGML/GGUF + Marlin)
• [x] FP8 KV Cache (CUDA & Metal, all backends)
• [x] TurboQuant KV Cache (2–4 bit compression)
• [x] FP8 Models (block-wise, SM90+)
• [x] MXFP4/NVFP4 Model Support
• [x] DeepSeek V3.2 and GLM-5.2 FP8 Model Support
• [x] MCP Integration & Tool Calling
• [x] Built-in ChatGPT-style Web UI

📚 References

• Python implementation: vllm-project
• vllm paper

Report Issue

If you encounter any problems, please create an issue.