boostr

What works

CI has no GPU. Some ops run on some backends only. docs/capabilities.md lists every capability, and docs/architecture.md holds the layout and design rules.

Quick Start

Install

Needs Rust 1.95 or newer.

[dependencies]
boostr = "0.11"                                    # CPU
# boostr = { version = "0.11", features = ["cuda"] }
# boostr = { version = "0.11", features = ["wgpu"] }

Basic Usage

Run flash attention on the CPU backend:

use boostr::ops::traits::attention::{AttnOutLayout, FlashAttentionOps};
use boostr::{CpuClient, CpuDevice, DType};
use boostr::RandomOps;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = CpuClient::new(CpuDevice::new());

    // Random tensors via numr's RandomOps: [batch, heads, seq, dim]
    let q = client.randn(&[1, 8, 32, 64], DType::F32)?;
    let k = client.randn(&[1, 8, 32, 64], DType::F32)?;
    let v = client.randn(&[1, 8, 32, 64], DType::F32)?;

    let (output, _lse) = client.flash_attention_fwd(
        &q,
        &k,
        &v,
        8,                         // num_heads
        8,                         // num_kv_heads
        64,                        // head_dim
        true,                      // causal
        0,                         // window_size (0 = no sliding window)
        None,                      // kv_seq_len
        None,                      // kv_start
        AttnOutLayout::HeadMajor,  // output layout: [B, H, S_q, D]
    )?;

    Ok(())
}

Load a GGUF Model

use boostr::format::Gguf;
use boostr::{CpuDevice, CpuRuntime};

// Open a GGUF file with buffered reads
let mut gguf = Gguf::open("model.gguf")?;
let device = CpuDevice::new();

// Load quantized tensors as QuantTensor and the others as f32
for name in gguf.tensor_names().map(|s| s.to_string()).collect::<Vec<_>>() {
    let info = gguf.tensor_info(&name)?;
    if info.ggml_type.is_quantized() {
        let qt = gguf.load_tensor_quantized::<CpuRuntime>(&name, &device)?;
    } else {
        let t = gguf.load_tensor_f32::<CpuRuntime>(&name, &device)?;
    }
}

Use Gguf::open_with_mmap(path, true) to memory-map the file. Each loader call releases mapped pages after it copies a tensor.

Paged KV Cache

use boostr::inference::PagedKvCache;
use boostr::{CpuClient, CpuDevice, CpuRuntime, DType, Tensor};
use boostr::RandomOps;

let device = CpuDevice::new();
let client = CpuClient::new(device.clone());
let (num_blocks, block_size, num_heads, head_dim) = (16, 16, 8, 64);

let cache = PagedKvCache::<CpuRuntime>::new(
    num_blocks, block_size, num_heads, head_dim, DType::F32, &device,
)?;

// key and value: [num_tokens, num_heads, head_dim]
let k = client.randn(&[4, num_heads, head_dim], DType::F32)?;
let v = client.randn(&[4, num_heads, head_dim], DType::F32)?;

// slot_mapping: [num_tokens], I64. Slot s maps to block s / block_size, offset s % block_size.
let slots = Tensor::<CpuRuntime>::from_slice(&[0i64, 1, 2, 3], &[4], &device)?;
cache.update(&k, &v, &slots, &client)?;

Build from Source

Copy .cargo/config.toml.example to .cargo/config.toml first. See CONTRIBUTING.md.

cargo build --release                  # CPU
cargo build --release --features cuda  # CUDA
cargo build --release --features wgpu  # WebGPU

• CUDA binaries: build.rs compiles kernels for the GPUs on the build host. Set NUMR_CUDA_ARCH=all for a binary that runs on another machine.
• boostr kernels only: BOOSTR_CUDA_ARCH overrides NUMR_CUDA_ARCH.

NUMR_CUDA_ARCH=all cargo build --release --features cuda

Feature Flags

Backends:

Precision:

Models and signal ops. No model feature implies another:

Tokenizer vocabularies:

Testing

# CPU tests, no model gates
cargo test

# Every model gate on the CPU backend
cargo test --features whisper,kokoro,neucodec,voxcpm,silero-vad,vits,clap,spacy-tagger,super-resolution,f16

# CUDA tests
cargo test --features cuda,f16

# Show test output
cargo test -- --nocapture

Run the format and lint checks in CONTRIBUTING.md before you open a pull request.

Ecosystem

boostr is part of the ml-rust organization:

• numr: Foundational numerical computing (tensors, ops, linalg, FFT)
• boostr: ML framework (this project)
• oxidizr: Training framework for Mamba2, MLA, and MoE (uses boostr)
• blazr: Inference server with an OpenAI-compatible API (uses boostr)
• compressr: Model quantization and compression (uses boostr)
• voxr: Audio and speech on top of boostr's models
• splintr: BPE tokenizer

Contributing

Contributions are welcome. See CONTRIBUTING.md for conventions and checks.

Questions and design discussion happen on Discord.

License

Licensed under the Apache License, Version 2.0. See LICENSE.

Parts of boostr port code from other open-source projects. Their copyright lines and licenses are in THIRD_PARTY_NOTICES.md.