pegainfer

Build from source

Use the Rust toolchain pinned in rust-toolchain.toml, a CUDA Toolkit with nvcc and cuBLAS, and a compatible NVIDIA driver. The default Qwen3 build needs no Python. Its driver floor is R545 / CUDA 12.3; newer toolkits and model-specific kernels can require a newer driver.

From the repository root, with the checkpoint downloaded:

export CUDA_HOME=/usr/local/cuda
cargo run --release -- --model-path models/Qwen3-4B

Always use --release for GPU builds. The server entrypoint is pegainfer-server; model crates contain the model implementation and diagnostics.

Feature builds and environment options

Qwen3.5 uses Triton AOT kernels, requiring Python and Triton at build time:

uv venv
uv pip install triton
export PEGAINFER_TRITON_PYTHON=.venv/bin/python
cargo run --release --features qwen35 -- --model-path models/Qwen3.5-4B

Qwen3.5 defaults to Triton even when a FlashInfer GDN candidate is linked. Explicit --qwen35-gdn-backend flashinfer-candidate selection requires SM120/Hv32/TP1 and fails on unsupported or unavailable candidates. Generation and production gates are described in the candidate README.

Other model lines have their own hardware and build requirements; follow the model guides below. Run cargo run --release -- --help for the compiled-in CLI.

Windows source builds

$env:CUDA_PATH = "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.x"
cargo run --release -p pegainfer-server -- --model-path models/Qwen3-4B

# Qwen3.5 additionally needs Triton at build time
uv venv .venv --python 3.12
uv pip install "triton-windows<3.7"
$env:PEGAINFER_TRITON_PYTHON = ".venv\Scripts\python.exe"
cargo run --release --features qwen35 -- --model-path models/Qwen3.5-4B

Performance

Selected serving measurements across dense, hybrid-attention, and MoE models. Each panel uses its own hardware, workload, and scale; the linked reports preserve the benchmark conditions.

Qwen3 serving footprint and additional measurements

Qwen3-4B on one RTX 5090, BF16, TP1: PegaInfer 70888b2 vs vLLM 0.24.0, loaded and serving the same model. PegaInfer is one process; the vLLM figure sums its process tree. This is a separate snapshot from the DSpark panel above.

The Qwen3 serving report also covers the 8B model, warm-prefix TTFT, host-tier restore, and the serving sweep against vLLM. Further reports cover DSpark vs matched DFlash, DFlash serving, and Gemma 4 12B long-context performance.

Supported Models

Only qwen3 is enabled by default, including in the prebuilt binary. Build other lines with --features <feature>. At launch, --model-path selects a checkpoint and its config.json identifies the model family.

Capabilities and maturity differ by model. Quantized formats are model-specific; the Qwen paths listed here use BF16. DeepSeek-V2-Lite's retained correctness and benchmark gates are documented separately from production readiness.

Architecture

Share the infrastructure; let each model own its execution. The frontend submits requests through an engine contract. Model schedulers decide how to batch work, manage state, and execute kernels on their target hardware.

The diagram shows ownership layers. Model engines select the components they need; cache integration and parallel strategies vary by model.

Editable diagram source

The step-based contract and legacy EngineHandle contract currently coexist. See the frontend architecture for the migration boundary and design direction for the reasoning behind model-owned engines.

API

Point an OpenAI-compatible client at http://localhost:8000/v1. Both /v1/completions and /v1/chat/completions support streaming.

curl -s http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"models/Qwen3-4B","prompt":"The capital of France is","max_tokens":32}'

curl -N http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"models/Qwen3-4B","messages":[{"role":"user","content":"Write a haiku about Rust."}],"max_tokens":64,"stream":true}'

The model guides document supported sampling fields and model-specific limits. Documentation covers the serving interface; metrics and dashboards cover observability.

Development

scripts/setup_dev.sh prepares a fresh NVIDIA Ubuntu host using the pinned Rust toolchain and build dependencies. CUDA must already be installed. Container development is documented in docker/README.md.

bash scripts/setup_dev.sh

Run checks in release mode; accuracy and integration tests need a GPU and model weights:

cargo test --release --workspace --lib

PEGAINFER_TEST_MODEL_PATH=models/Qwen3-4B \
  cargo test --release -p pegainfer-qwen3 --test hf_golden_gate

PEGAINFER_TEST_MODEL_PATH=models/Qwen3.5-4B \
  cargo test --release -p pegainfer-qwen35 --features qwen35 --lib executor::hf_golden_gate -- --test-threads=1

PEGAINFER_TEST_MODEL_PATH=models/Qwen3.5-4B \
  cargo test --release -p pegainfer-qwen35 --features qwen35 --lib scheduler::e2e_tests -- --test-threads=1

Browse the engineering docs index for model-specific gates, profiling, and benchmark methodology.

The getting started guide and model pages live at pegainfer.org. Read the engineering stories at pegainfer.org/blog: weight loading, speculative decoding, CUDA graph export, and prefill/decode overlap.

License

Apache-2.0. See NOTICE for third-party attributions. The Dynamo-derived kvbm/kvbm-logical crate retains its original Apache-2.0 headers and NVIDIA Dynamo notice.