llama-cpp-rs
Library Featured
llama.cpp bindings for Rust
- Downloads
- 341,382
- Dependents
- 98
- Stars
- 665
- Forks
- 261
- Contributors
- 101
- Commits
- 1,925
- Latest release
- 0.1.161 ·
About
This is the home for llama-cpp-2. It also contains the llama-cpp-sys bindings which are updated semi-regularly and in sync with llama-cpp-2.
This project was created with the explict goal of staying as up to date as possible with llama.cpp, as a result it is dead simple, very close to raw bindings, and does not follow semver meaningfully.
Check out the docs.rs for crate documentation or the readme for high level information about the project.
Try it
We maintain a super simple example of using the library:
Clone the repo
git clone --recursive https://github.com/utilityai/llama-cpp-rs
cd llama-cpp-rs
Run the simple example (add --features cuda if you have a cuda gpu)
cargo run --release --bin simple -- --prompt "The way to kill a linux process is" hf-model TheBloke/Llama-2-7B-GGUF llama-2-7b.Q4_K_M.gguf
Output
Read the full README
ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
llama_model_params { n_gpu_layers: 1000, split_mode: 1, main_gpu: 0, tensor_split: 0x0, progress_callback: None, progress_callback_user_data: 0x0, kv_overrides: 0x0, vocab_only: false, use_mmap: true, use_mlock: false }
llama_model_loader: loaded meta data with 19 key-value pairs and 291 tensors from /home/marcus/.cache/huggingface/hub/models--TheBloke--Llama-2-7B-GGUF/snapshots/b4e04e128f421c93a5f1e34ac4d7ca9b0af47b80/llama-2-7b.Q4_K_M.gguf (version GGUF V2)
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = LLaMA v2
llama_model_loader: - kv 2: llama.context_length u32 = 4096
llama_model_loader: - kv 3: llama.embedding_length u32 = 4096
llama_model_loader: - kv 4: llama.block_count u32 = 32
llama_model_loader: - kv 5: llama.feed_forward_length u32 = 11008
llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 7: llama.attention.head_count u32 = 32
llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 32
llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 10: general.file_type u32 = 15
llama_model_loader: - kv 11: tokenizer.ggml.model str = llama
llama_model_loader: - kv 12: tokenizer.ggml.tokens arr[str,32000] = ["", "", "", "<0x00>", "<...
llama_model_loader: - kv 13: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 14: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv 15: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 16: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 17: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 18: general.quantization_version u32 = 2
llama_model_loader: - type f32: 65 tensors
llama_model_loader: - type q4_K: 193 tensors
llama_model_loader: - type q6_K: 33 tensors
llm_load_vocab: special tokens definition check successful ( 259/32000 ).
llm_load_print_meta: format = GGUF V2
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 32000
llm_load_print_meta: n_merges = 0
llm_load_print_meta: n_ctx_train = 4096
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 32
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 1
llm_load_print_meta: n_embd_k_gqa = 4096
llm_load_print_meta: n_embd_v_gqa = 4096
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff = 11008
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 4096
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: model type = 7B
llm_load_print_meta: model ftype = Q4_K - Medium
llm_load_print_meta: model params = 6.74 B
llm_load_print_meta: model size = 3.80 GiB (4.84 BPW)
llm_load_print_meta: general.name = LLaMA v2
llm_load_print_meta: BOS token = 1 ''
llm_load_print_meta: EOS token = 2 ''
llm_load_print_meta: UNK token = 0 ''
llm_load_print_meta: LF token = 13 '<0x0A>'
llm_load_tensors: ggml ctx size = 0.22 MiB
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors: CUDA0 buffer size = 3820.94 MiB
llm_load_tensors: CPU buffer size = 70.31 MiB
..................................................................................................
Loaded "/home/marcus/.cache/huggingface/hub/models--TheBloke--Llama-2-7B-GGUF/snapshots/b4e04e128f421c93a5f1e34ac4d7ca9b0af47b80/llama-2-7b.Q4_K_M.gguf"
llama_new_context_with_model: n_ctx = 2048
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CUDA0 KV buffer size = 1024.00 MiB
llama_new_context_with_model: KV self size = 1024.00 MiB, K (f16): 512.00 MiB, V (f16): 512.00 MiB
llama_new_context_with_model: CUDA_Host input buffer size = 13.02 MiB
ggml_gallocr_reserve_n: reallocating CUDA0 buffer from size 0.00 MiB to 164.01 MiB
ggml_gallocr_reserve_n: reallocating CUDA_Host buffer from size 0.00 MiB to 8.00 MiB
llama_new_context_with_model: CUDA0 compute buffer size = 164.01 MiB
llama_new_context_with_model: CUDA_Host compute buffer size = 8.00 MiB
llama_new_context_with_model: graph splits (measure): 3
n_len = 32, n_ctx = 2048, k_kv_req = 32
The way to kill a linux process is to send it a SIGKILL signal.
The way to kill a windows process is to send it a S
decoded 24 tokens in 0.23 s, speed 105.65 t/s
load time = 727.50 ms
sample time = 0.46 ms / 24 runs (0.02 ms per token, 51835.85 tokens per second)
prompt eval time = 68.52 ms / 9 tokens (7.61 ms per token, 131.35 tokens per second)
eval time = 225.70 ms / 24 runs (9.40 ms per token, 106.34 tokens per second)
total time = 954.18 ms
Hacking
Ensure that when you clone this project you also clone the submodules. This can be done with the following command:
git clone --recursive https://github.com/utilityai/llama-cpp-rs
or if you have already cloned the project you can run:
git submodule update --init --recursive
Compatibility
- Runs on
- CUDA, WebGPU, Metal, ROCm, Vulkan
What shows it
- CUDA: llama-cpp-2 feature cuda, llama-cpp-sys-2 feature cuda
- WebGPU: llama-cpp-2 feature webgpu, llama-cpp-sys-2 feature webgpu
- Metal: llama-cpp-2 feature metal, llama-cpp-sys-2 feature metal
- ROCm: llama-cpp-2 feature rocm, llama-cpp-sys-2 feature rocm
- Vulkan: llama-cpp-2 feature vulkan, llama-cpp-sys-2 feature vulkan
- Native code
- Required
- Binds native code: llama
- Build uses a native toolchain: bindgen, cc, cmake
- Python bindings
- No Python binding crate found
- License
- Apache-2.0 OR MIT
Compare with similar projects
Order: strongest tier first, then most downloads, then name, with unmeasured values last. A library counts downloads over 30 days. An app counts binary downloads per release. An app with none counts its crate downloads over 30 days. ML Rust runs this catalogue, so its closest project always gets a row.
| Project | Downloads | Dependents | Stars | Latest release | Runs on (GPU) | Native code | License |
|---|---|---|---|---|---|---|---|
|
llama.cpp bindings for Rust |
341,382 downloads over 30 days | 98 | 665 | 0.1.161 | CUDA, WebGPU, Metal, ROCm, Vulkan | Binds native code: llama. Build uses a native toolchain: bindgen, cc, cmake | Apache-2.0 OR MIT |
|
ort Featured
A safe Rust wrapper for ONNX Runtime 1.31 - Optimize and accelerate machine learning inference & training |
3,902,768 downloads over 30 days | 694 | 2,611 | 2.0.0-rc.15 | CUDA, WebGPU, ROCm | Binds native code: onnxruntime, ort-sys. Build uses a native toolchain: pkg-config (feature pkg-config) | Apache-2.0 OR MIT |
|
ctranslate2-rs Listed
Rust bindings for OpenNMT/CTranslate2 |
10,244 downloads over 30 days | 6 | 63 | 0.10.4 | CUDA | Binds native code: cxx, link-cplusplus, intel-onemkl-prebuild (feature mkl), onednn-src (feature dnnl), openblas-src (feature openblas) and 1 more. Build uses a native toolchain: cmake, cxx-build | MIT |
|
llama-gguf Listed
A high-performance Rust implementation of llama.cpp - LLM inference engine with full GGUF support |
1,787 downloads over 30 days | 3 | 21 | 0.14.0 | CUDA, Metal, Vulkan | Binds native code: hailort-sys (feature hailo), libsqlite3-sys (feature rag-sqlite) | Apache-2.0 OR MIT |
|
Rust bindings for TensorFlow Lite |
550 downloads over 30 days | 1 | 129 | 0.9.8 | GPU support unknown | Build uses a native toolchain: bindgen | Apache-2.0 OR MIT |
|
Inference server for LLMs, speech and embeddings: OpenAI-compatible routes with streaming, plus speech synthesis (TTS) and transcription (ASR). Serves Llama, Mistral, Qwen, Mamba2, MLA+MoE and hybrid models from SafeTensors, AWQ, GPTQ and GGUF. |
8 binary downloads per release | Not applicable | 7 | 0.7.1 | CUDA, WebGPU | Binds native code: ring (feature distributed) | Apache-2.0 |
|
baseRT New
Safe Rust bindings for the BaseRT LLM inference engine (Apple Silicon) |
49 downloads over 30 days | 2 | 247 | 0.4.6 | GPU support unknown | Binds native code: baseRT, baseRT-sys | Apache-2.0 |
Why FeaturedIt passes every Featured gate.
What it meets
- It has 63 active dependents and 59 new dependents in 365 days. Featured needs 11 active dependents or 9 new dependents in 365 days.
- Active dependents are at 100% of their peak. Featured needs at least 50%.
- Docs are present. The release commit or a manifest ties the crate to the repository.
- No vulnerability advisory stays unpatched past the grace window. At least one version is neither a prerelease nor 0.0.x.
- The last activity is on 4 Nov 2026, 5 days ago. The rules expect an update within 1800 days. Dormancy rule
Exact rules: Featured library. The check on applied thresholds version 9.
Every rule outcome, as the check records it
expected-interval: version 0.1.161 base 450, doubled for 341382 downloads per month, doubled for 63 active users: E = 1800 daysdormancy: last activity 2026-11-04, 5 days ago, E = 1800: activefeatured: featured: active_users 63, new_adopters 59, former_glory 1
Crates2 crates
- llama-cpp-2 Primary
- llama-cpp-sys-2
Measurements and sources
Each signal row shows the raw value, its source and the fetch date. The Sources list gathers every distinct source on this page. Movement needs more history.
Sources6 sources
- Description: the crates.io description
- crates.io database dump, fetched
- RustSec advisory database, fetched
- GitHub repository, fetched
- crates.io package, fetched
- GitHub contributors, fetched
Adoption63 active dependents, 59 new dependents in 365 days1 not measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Active dependents | 63 | crates.io database dump | Yes | |
| New dependents (365 days) | 59 | crates.io database dump | Yes | |
| Peak active dependents | 62 | crates.io database dump | Yes | |
| Active dependents as a share of the peak | 1 | crates.io database dump | Yes | |
| Binary downloads per release | Not measured | Yes |
MaintenanceLatest human commit: 5 days ago3 not measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Latest human commit | 2026-11-04 | GitHub repository | Yes | |
| Latest release | 2026-11-03 | crates.io database dump | Yes | |
| First release | 2024-01-02 | crates.io database dump | Yes | |
| Latest version | 0.1.161 | crates.io database dump | Yes | |
| Days the oldest outside issue waits for a reply | 355 | GitHub repository | Yes | |
| Maintainer replies over 365 days | 3 | GitHub repository | Yes | |
| Repository archived | false | GitHub repository | Yes | |
| Repository gone | false | GitHub repository | Yes | |
| Release commit or manifest in the repo | true | crates.io package | Yes | |
| Published docs | true | crates.io database dump | Yes | |
| Latest release fails cargo check | Not measured | Yes | ||
| Build check date | Not measured | Yes | ||
| Build check toolchain | Not measured | Yes |
SecurityUnpatched vulnerabilities: none
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Unpatched vulnerabilities past the grace window | none | RustSec advisory database | Yes | |
| RustSec unmaintained advisory | none | RustSec advisory database | Yes | |
| Deprecation notice | none | crates.io database dump | Yes | |
| Deprecated requirement | none | crates.io database dump | Yes |
Distribution139 releases, 341,382 downloads over 30 days5 not measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Releases | 139 | crates.io database dump | Yes | |
| Downloads over 30 days | 341382 | crates.io database dump | Yes | |
| Binary downloads over the last 3 releases | Not measured | Yes | ||
| Homebrew installs on request over 365 days | Not measured | Yes | ||
| Repology repository families | Not measured | Yes | ||
| Docker pull growth over 90 days | Not measured | Yes | ||
| Docker pulls at the growth baseline | Not measured | Yes | ||
| Every version a prerelease | false | crates.io database dump | Yes | |
| Every version 0.0.x | false | crates.io database dump | Yes | |
| Every version yanked | false | crates.io database dump | Yes | |
| Crate deleted from crates.io | false | crates.io database dump | Yes |
Repository665 stars, 261 forks, 101 contributors, 1,925 commits. Stars alone never qualify a project.
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| GitHub stars | 665 | GitHub repository | Yes — one part of the repository route | |
| GitHub forks | 261 | GitHub repository | Yes — one part of the repository route | |
| Contributors with a GitHub account | 101 | GitHub contributors | Yes — one part of the repository route | |
| Commits on the default branch | 1925 | GitHub repository | No | |
| Repository's main crate left out of the entry | none | crates.io database dump | Yes |
Facts that never count
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| License | Apache-2.0 OR MIT | crates.io database dump | No | |
| Runs on, from listed crates and their direct dependencies | CUDA, WebGPU, Metal, ROCm, Vulkan | crates.io database dump | No | |
| What shows each backend | CUDA: llama-cpp-2 feature cuda, llama-cpp-sys-2 feature cuda; WebGPU: llama-cpp-2 feature webgpu, llama-cpp-sys-2 feature webgpu; Metal: llama-cpp-2 feature metal, llama-cpp-sys-2 feature metal; ROCm: llama-cpp-2 feature rocm, llama-cpp-sys-2 feature rocm; Vulkan: llama-cpp-2 feature vulkan, llama-cpp-sys-2 feature vulkan | crates.io database dump | No | |
| Native code, from listed crates and their dependencies | Binds native code: llama. Build uses a native toolchain: bindgen, cc, cmake | crates.io database dump | No | |
| Python bindings, from direct dependencies of listed crates | No Python binding crate found | crates.io database dump | No | |
| Crates that depend on a listed crate directly | 98 | crates.io database dump | No |
Corrections and review history
No correction applies to this entry.
Judgments
A judgment enters only through an approved issue and a merged pull request.
No judgment applies to this entry.