compressr

Roadmap. Each capability below has an enum variant. compressr refuses each one before it creates an output file:

The optimize subcommand takes --method only. It always errors and writes no file. Every method it names (GPTQ, AWQ, EXL2, NF4, FP8) is on the roadmap. A flat GGUF method passed to it is refused with the convert command that writes it.

SafeTensors output is never quantized. Block quantization writes GGUF, or TCF with --format tcf.

TCF comes from a full-precision checkpoint only. A GGUF or TCF input is refused for TCF output.

Quick Start

compressr needs a Rust toolchain. Cargo.toml takes boostr from crates.io. A checkout beside unreleased sibling crates needs the local build overrides first. The file is gitignored.

cp .cargo/config.toml.example .cargo/config.toml
cargo install --path .

compressr runs every boostr kernel on the CPU. It has no GPU path.

Convert a checkpoint to a quantized GGUF file:

compressr convert ./llama-3-8b --format gguf --quantization q4_k_m

Cast a checkpoint to BF16 SafeTensors:

compressr convert ./llama-3-8b --format safetensors --dtype bf16

Inspect a checkpoint:

compressr info ./llama-3-8b            # format, tensor count, size, metadata
compressr info ./llama-3-8b --verbose  # one line per tensor
compressr info ./llama-3-8b --summary  # tensor count and size only

Usage

Quantization

--quantization takes these flat formats: q4_0, q4_1, q8_0, q2_k, q3_k, q4_k, q5_k, q6_k, iq4_nl, and iq4_xs. It also takes the lowbit repack names pq2_0 and ptq1_0.

A mixed-precision name assigns different GGUF block formats to different tensors. The mix names are q4_k_s, q4_k_m, q5_k_s, and q5_k_m. The mix reproduces llama.cpp's per-tensor allocation by role and layer.

--quantization and --dtype are mutually exclusive. Quantization already fixes the stored representation.

Measured sensitivity and importance matrices

--sensitivity FILE reads measured task deltas as JSON lines. It requires a mixed-precision --quantization and --format gguf or --format tcf. The promotions of the mix are ranked by measured damage per element. They spend the same bytes that llama.cpp's rules spend on the checkpoint. Tensors whose rows are not a whole number of 256-element super-blocks cannot be promoted. compressr counts and reports them.

--imatrix FILE reads an importance matrix from boostr's imatrix example. It weights each quantizer choice by the mean squared activation of each input column. A conversion without the flag writes the same bytes it always wrote. --imatrix works with --format gguf or --format tcf, and with K-quants, iq4_nl, iq4_xs, and K-quant mixes. A tensor the matrix does not measure is quantized without weights.

compressr convert ./llama-3-8b --format tcf --quantization q4_k_m --sensitivity llama.sens.jsonl
compressr convert ./llama-3-8b --format gguf --quantization q4_k_m --imatrix llama.imatrix

Lowbit repack

compressr convert model-PTQ1_0.gguf --format gguf --quantization pq2_0
compressr convert model-PQ2_0.gguf --format gguf --quantization ptq1_0

PTQ1_0 and PQ2_0 store the same ternary codes in two block layouts. A repack moves the codes and scales of each source tensor, and it rounds nothing. Tensors already in the target format, float norms, and other types pass through. A PQ2_0 block that holds code 3 has no trit, so ptq1_0 refuses it. The error names the tensor and block. Neither format is quantized from float weights. A SafeTensors or F16 input is refused.

Output shards

compressr convert ./llama-3-8b --format safetensors --dtype bf16 --max-shard-size 2GB
compressr convert ./llama-3-8b --format safetensors --dtype bf16 --max-shard-size none

SafeTensors output over the limit becomes model-00001-of-000NN.safetensors shards plus model.safetensors.index.json. Output under the limit stays one model.safetensors with no index. A tensor is never split, so a tensor larger than the limit gets a shard of its own. The flag takes a byte count, a decimal suffix (5GB, 500MB), or a binary suffix (1GiB, 512MiB). The default 5GB matches HuggingFace. GGUF ignores the flag.

Input resolution

A directory with a model.safetensors.index.json loads exactly the shards that index names. Other .safetensors files beside them, such as a LoRA adapter or an earlier conversion's output, are ignored.

A directory with shards and no index loads every .safetensors file in name order.

GGUF headers from HuggingFace checkpoints

For a HuggingFace checkpoint, compressr writes the GGUF header that convert_hf_to_gguf.py writes. The architecture comes from model_type in config.json, then architecture, then the first entry of architectures. Pass --arch when the checkpoint names none.

• Hyperparameters: context length, embedding length, block count, feed-forward length, head counts, and the norm epsilon. It also writes .rope.* keys with scaling, expert counts, and key and value lengths when head_dim is stated. A key whose config.json field is absent is omitted.
• RoPE tensors: llama3 scaling writes rope_freqs.weight. LongRoPE writes rope_factors_long.weight and rope_factors_short.weight.
• Tokenizer: tokenizer.model gives tokenizer.ggml.model = "llama". A byte-level BPE tokenizer.json gives "gpt2", with merges and a tokenizer.ggml.pre name from llama.cpp's fingerprint table. An unknown fingerprint omits pre and logs a warning. Special ids, add-token flags, and tokenizer.chat_template follow llama.cpp.
• Q/K order: llama.cpp's llama architecture rotates adjacent pairs. compressr reorders q_proj and k_proj rows to match. boostr's GGUF loader restores them.
• HuggingFace names: the architectures in the last row keep HuggingFace tensor names. llama.cpp's converter changes their values: Gemma norms, Phi-3 fused projections, and merged MoE experts. llama.cpp cannot load those files.

Options

compressr convert <INPUT>

compressr info <MODEL>

Memory

Opening a model reads its headers only. Each tensor is then read into an owned buffer, converted, written, and dropped before the next one starts. Peak resident memory is roughly the largest tensor plus its converted copy. It depends on the widest layer, not on the model size.

The loader copies tensor bytes out of the file. It does not memory-map them.

Architecture

compressr is a thin layer over boostr. Every quantization kernel and block layout lives in boostr, behind its Runtime trait.

A tensor that cannot be block-quantized is stored as F32 instead of failing the run. A rank-1 tensor has no row to pack blocks along. A row length that is not a multiple of the block size cannot be packed. llama.cpp makes the same choice. The conversion report counts both cases.

Contributing

Contributions are welcome. See CONTRIBUTING.md for conventions and checks.

Questions and design discussion happen on Discord.

License

Licensed under Apache-2.0. See LICENSE.