blazr

What works

CI runs no GPU test. It compile-checks wgpu and does not build cuda.

Quick Start

Install a prebuilt archive

• Every archive above includes the audio feature.
• Android and iOS archives hold libblazr.a. See C library.
• Archive names carry no version, so the latest release keeps the same URL.
• docs/deployment.md lists every archive.

curl -LO https://github.com/ml-rust/blazr/releases/latest/download/blazr-linux-x64.tar.gz
curl -LO https://github.com/ml-rust/blazr/releases/latest/download/SHA256SUMS
sha256sum --check --ignore-missing SHA256SUMS
tar xzf blazr-linux-x64.tar.gz
./blazr --help

On macOS, use shasum -a 256 --check --ignore-missing SHA256SUMS. docs/deployment.md has the Windows steps.

Install with cargo

Needs Rust 1.95 or newer.

cargo install blazr --locked --features audio          # CPU
cargo install blazr --locked --features audio,cuda     # NVIDIA CUDA

Run a model

blazr run, generate, chat, and bench take a model path. They never fetch a model. blazr pull downloads one first.

blazr pull Qwen/Qwen2.5-0.5B      # writes ./models/Qwen2.5-0.5B
blazr generate --model ./models/Qwen2.5-0.5B --prompt "Once upon a time" --max-tokens 100
blazr run ./models/Qwen2.5-0.5B   # interactive REPL

blazr pull writes to --output, then BLAZR_MODEL_DIR, then the model_dir config key, then ./models.

Start the server

blazr serve --model takes a path, or a name that blazr list shows.

blazr serve --model Qwen2.5-0.5B --port 8080

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100,
    "stream": true
  }'

• blazr serve does not download LLM weights. Run blazr pull first.
• blazr serve fetches only Kokoro's G2P files, on first use.
• Without --model, the server loads the default_model key when set. Otherwise it starts with no model.
• Requests load the named model on first use. See docs/deployment.md.

Build from source

• Backends: cuda takes priority over wgpu when both are on.
• Devices: the device config key takes cpu, cuda, cuda:N, wgpu, and wgpu:N. --cpu forces the CPU where a command offers it.
• CUDA binaries: set NUMR_CUDA_ARCH=all for a binary that runs on another machine.
• Tensor parallelism: needs NCCL. A degree above 1 fails in any other build. It is tested on one GPU only. See docs/tuning.md.
• Library use: --no-default-features builds the engine with no CLI or HTTP stack. The blazr binary needs the default cli and server.
• Commands by feature: docs/architecture.md.

Supported Models

The Evidence column covers the forward pass on a raw prompt. It says nothing about chat. tests/chat_stop_reference.rs checks chat prompt ids and stop ids against transformers for Qwen2.5, Qwen3, Gemma 2, and Phi-3.5, and one real Qwen3 turn that ends on <|im_end|>.

A base checkpoint is not a chat model. Qwen2.5-0.5B and gemma-2-2b never learned their turn markers. Through /v1/chat/completions they run past the first sentence and print control text where a marker belongs, and transformers prints the same kind of control text from the same prompt. Use an -Instruct, -it, or chat checkpoint for chat.

docs/models.md holds the full matrix, with an evidence level per cell.

Model Formats

blazr detects the format from checkpoint contents.

model_dir/                         # SafeTensors (HuggingFace)
├── model.safetensors              # or sharded: model-00001-of-00002.safetensors
├── config.json                    # HuggingFace model config
└── tokenizer_config.json          # Chat template (optional)

checkpoint_dir/                    # SafeTensors (oxidizr)
├── model.safetensors
└── config.json                    # oxidizr config (optional, inferred from tensors)

model.gguf                         # GGUF: weights, tokenizer, and config in one file

GGUF quantization formats run from Q2_K through Q8_0, plus the IQ and TQ series. docs/models.md lists the CPU and CUDA kernel for each format.

GGUF files load through mmap on every target except wasm32. A failed map falls back to file reads.

API Reference

This table lists the main routes. docs/api.md lists every route, with its request fields, responses, and errors.

CLI Reference

Run blazr <command> --help for every flag. docs/configuration.md documents the flags, the config file, and the environment variables.

Chat Templates and Images

Chat routes render messages with the model's own Jinja template. --chat-template replaces it with a built-in name (chatml, llama3, mistral, phi3, gemma, deepseek, generic) or a template file. See docs/models.md.

With --mmproj, the chat, responses, and messages routes accept images beside text. Without it, an image request gets a 400 that names the flag. See docs/api.md.

Speech

Build with --features audio, or use a desktop archive. The feature adds /v1/audio/* routes to blazr serve and three commands.

The commands write reports to standard output and logs to standard error.

A command fetches a missing model from the Hugging Face Hub on first use. docs/models.md holds the model list, search path, and device rules.

Use as a library

The engine builds without the CLI and the HTTP stack:

[dependencies]
blazr = { version = "0.7", default-features = false, features = ["f16"] }

Scheduler loads models from a directory on first use. Executor::generate streams tokens and stops when you drop the stream. This example needs anyhow, futures, and an async runtime such as tokio.

use std::pin::pin;

use blazr::boostr::{CpuDevice, CpuRuntime};
use blazr::tokenizer::Tokenize;
use blazr::{GenerationConfig, Scheduler};
use futures::StreamExt;

async fn complete(on_token: impl Fn(&str)) -> anyhow::Result<()> {
    let scheduler = Scheduler::<CpuRuntime>::new("./models".into(), CpuDevice::new());
    let executor = scheduler.get_executor("llama-3.2-1b").await?;
    let prompt = executor.tokenizer().encode("The capital of France is");
    let config = GenerationConfig {
        max_tokens: 64,
        ..Default::default()
    };
    let mut tokens = pin!(executor.generate(&prompt, &config));
    while let Some(token) = tokens.next().await {
        on_token(&token?.text);
    }
    Ok(())
}

docs.rs/blazr documents the API.

C library

The capi feature builds a C library for app embedders.

cargo build --release --lib --no-default-features --features capi,f16

• Output: libblazr.a and libblazr.so in target/release/.
• Header: include/blazr.h.
• Status: every call returns a BlazrStatus. Zero and above mean success.
• Bindings: blazr ships no Kotlin or Swift bindings. Call the C functions through JNI or a bridging header.
• Functions and link steps: examples/capi/README.md. examples/capi/smoke.c calls every step.

Decision models

blazr serves System One decision models on POST /v1/systemone. A decision model answers typed questions about one state. It returns probabilities and does not generate text.

BLAZR_MODEL_DIR=/path/to/models blazr serve --model decider-4b --port 8080

Limits:

• Clef-Flash memory: the text backbone takes 16.7 GiB in BF16. It does not fit a 12 GB GPU.
• CUDA: Kev and Clef are not tested on it.
• Clef 27B: not tested and not claimed.
• Other families: not served.

docs/decision-models.md has run commands, family detection, calibration, and the tested matrix.

Documentation

Related Projects

• boostr: ML framework, model architectures, quantization
• numr: tensors, linear algebra, CPU and GPU backends
• voxr: audio and speech pipelines behind audio
• oxidizr: training framework for hybrid architectures
• splintr: BPE tokenizer
• compressr: model conversion and compression

Contributing

Contributions are welcome. See CONTRIBUTING.md for conventions and checks.

Questions and design discussion happen on Discord.

License

Licensed under Apache-2.0. See LICENSE.

Built with Llama.

blazr embeds the Llama 3 tokenizer vocabulary through splintr. That vocabulary is under the Llama 3.1 Community License.

Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.

Each release archive carries that licence as LICENSE-LLAMA, and the licence of every compiled-in crate in THIRD-PARTY-LICENSES.txt.