sentencepiece-rs
Library Listed
Rust runtime reimplementation of SentencePiece model loading, normalization, encoding, and decoding.
- Downloads
- 30,659
- Dependents
- 5
- Stars
- 2
- Forks
- 1
- Contributors
- 2
- Commits
- 8
- Latest release
- 0.2.2 ·
About
SentencePiece model loading, normalization, encoding, and decoding in pure Rust. Load a .model file, tokenize text, and decode tokens back into text.
Install
[dependencies]
sentencepiece-rs = "0.2"
Quick Start
use sentencepiece_rs::SentencePieceProcessor;
fn main() -> sentencepiece_rs::Result<()> {
let sp = SentencePieceProcessor::open("tokenizer.model")?;
let pieces = sp.encode("hello rust world")?;
let ids = sp.encode_to_ids("hello rust world")?;
let text = sp.decode_ids(&ids)?;
println!("{pieces:?}");
println!("{ids:?}");
println!("{text}");
Ok(())
}
Decode Pieces
use sentencepiece_rs::SentencePieceProcessor;
let sp = SentencePieceProcessor::open("tokenizer.model")?;
let text = sp.decode(&["▁hello", "▁world"])?;
assert_eq!(text, "hello world");
# Ok::<(), sentencepiece_rs::Error>(())
Extra Options
use sentencepiece_rs::SentencePieceProcessor;
let mut sp = SentencePieceProcessor::open("tokenizer.model")?;
sp.set_encode_extra_options("bos:eos")?;
let ids = sp.encode_to_ids("ship it")?;
# Ok::<(), sentencepiece_rs::Error>(())
Supported options: bos, eos, reverse, unk_piece.
Read the full README
Compatibility Notes
The crate reads standard SentencePiece model protobufs and uses the embedded normalizer trie, so normal .model files should load without drama. The runtime behavior follows the C++ implementation, but this is not a line-by-line rewrite.
The big missing chunk is training. Bring your own existing model for now.
Documentation
References
- Original SentencePiece repo: google/sentencepiece
- Paper: arXiv PDF
Test Locally
cargo fmt --all
cargo test --all-features
cargo clippy --all-targets --all-features -- -D warnings
cargo doc --all-features --no-deps
License and Terms of Use
This crate is licensed under the Apache License, Version 2.0. See ./LICENSE.
The original SentencePiece source is also licensed under the Apache License, Version 2.0. This crate is a Rust reimplementation that uses the upstream source as the behavioral reference; it is not a C++ binding and does not link against the C++ library.
Apache 2.0 is permissive. In normal-human terms, you can use, modify, distribute, sublicense, and ship derivative work, including commercially, as long as you follow the license terms.
If you redistribute this crate, the upstream source, or a modified version, keep the important bits intact:
- include a copy of the Apache 2.0 license
- preserve copyright, patent, trademark, and attribution notices that apply
- mark modified files when you change Apache-licensed source
- include upstream
NOTICEcontent if a distributed upstream package includes one - do not imply Google or SentencePiece trademark endorsement
The software is provided as-is, without warranty, and the Apache 2.0 patent grant terminates if you sue over patent infringement involving the licensed work. That is the deal. Pretty reasonable, honestly.
This README is a practical summary, not legal advice. The actual license text in ./LICENSE and the upstream SentencePiece license are the source of truth.
Compatibility
- Runs on
- GPU support unknown
What shows it
- no dependency or feature matches a backend row
- Native code
- None found
- No native code found
- Python bindings
- No Python binding crate found
- License
- Apache-2.0
Compare with similar projects
Order: strongest tier first, then most downloads, then name, with unmeasured values last. A library counts downloads over 30 days. An app counts binary downloads per release. An app with none counts its crate downloads over 30 days. ML Rust runs this catalogue, so its closest project always gets a row.
| Project | Downloads | Dependents | Stars | Latest release | Runs on (GPU) | Native code | License |
|---|---|---|---|---|---|---|---|
|
Rust runtime reimplementation of SentencePiece model loading, normalization, encoding, and decoding. |
30,659 downloads over 30 days | 5 | 2 | 0.2.2 | GPU support unknown | No native code found | Apache-2.0 |
|
tiktoken-rs Featured
Library for encoding and decoding with the tiktoken library in Rust |
3,939,182 downloads over 30 days | 449 | 410 | 0.12.1 | GPU support unknown | No native code found | MIT |
|
whatlang-rs Featured
Fast and lightweight language identification library for Rust. |
882,019 downloads over 30 days | 74 | 1,087 | 0.18.0 | GPU support unknown | No native code found | MIT |
|
text-splitter Featured
Split text into semantic chunks, up to a desired chunk size. Supports calculating length by characters and tokens, and is callable from Rust and Python. |
271,174 downloads over 30 days | 56 | 645 | 0.33.1 | GPU support unknown | Binds native code: tree-sitter (feature code), tree-sitter-language (feature code) | MIT |
|
spm_precompiled Listed
This crate aims to emulate https://github.com/google/sentencepiece Dart::DoubleArray struct and it's Normalizer. This crate is highly specialized and not intended for general use. |
6,527,079 downloads over 30 days | 7 | 23 | 0.1.4 | GPU support unknown | No native code found | Apache-2.0 |
|
splintr Listed
Tokenizer for Rust and Python: pure Rust by default, with 18 bundled vocabularies and four backends covering BPE, SentencePiece and WordPiece. Its output matches Hugging Face tokenizers id for id on 14 model families, including GPT-2, Qwen, Whisper, T5 and BERT. Python bindings |
8,204 downloads over 30 days | 7 | 63 | 0.22.2 | GPU support unknown | Binds native code: pcre2-sys (feature pcre2) | MIT |
Why ListedNo archive, New or Featured rule places it, so it is Listed.
What it lacks for Featured
- It has 6 active dependents and 6 new dependents in 365 days. Featured needs 11 active dependents or 9 new dependents in 365 days. The repository route reads 2 stars, 1 fork and 2 contributors. It needs 4,539 stars, 434 forks and 79 contributors together.
What else the check found
- The last activity is on 20 May 2026, 173 days ago. The rules expect an update within 365 days. Dormancy rule
Exact rules: Listed, Featured library. The check on applied thresholds version 9.
Every rule outcome, as the check records it
expected-interval: version 0.2.2 base 300, capped at 4 releases = 200, raised to the 365-day floor: E = 365 daysdormancy: last activity 2026-05-20, 173 days ago, E = 365: activefeatured: not featured: active_users 6 < 11 and new_adopters 6 < 9 and repository route stars 2 (needs 4539), forks 1 (needs 434), contributors 2 (needs 79)listed: listed: no archive, new or featured rule applies
Measurements and sources
Each signal row shows the raw value, its source and the fetch date. The Sources list gathers every distinct source on this page. Movement needs more history.
Sources6 sources
- Description: the crates.io description
- crates.io database dump, fetched
- RustSec advisory database, fetched
- GitHub repository, fetched
- crates.io package, fetched
- GitHub contributors, fetched
Adoption6 active dependents, 6 new dependents in 365 days1 not measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Active dependents | 6 | crates.io database dump | Yes | |
| New dependents (365 days) | 6 | crates.io database dump | Yes | |
| Peak active dependents | 5.5 | crates.io database dump | Yes | |
| Active dependents as a share of the peak | 1 | crates.io database dump | Yes | |
| Binary downloads per release | Not measured | Yes |
MaintenanceLatest human commit: 5 months ago3 not measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Latest human commit | 2026-05-20 | GitHub repository | Yes | |
| Latest release | 2026-05-20 | crates.io database dump | Yes | |
| First release | 2026-05-10 | crates.io database dump | Yes | |
| Latest version | 0.2.2 | crates.io database dump | Yes | |
| Days the oldest outside issue waits for a reply | 0 | GitHub repository | Yes | |
| Maintainer replies over 365 days | 1 | GitHub repository | Yes | |
| Repository archived | false | GitHub repository | Yes | |
| Repository gone | false | GitHub repository | Yes | |
| Release commit or manifest in the repo | true | crates.io package | Yes | |
| Published docs | true | crates.io database dump | Yes | |
| Latest release fails cargo check | Not measured | Yes | ||
| Build check date | Not measured | Yes | ||
| Build check toolchain | Not measured | Yes |
SecurityUnpatched vulnerabilities: none
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Unpatched vulnerabilities past the grace window | none | RustSec advisory database | Yes | |
| RustSec unmaintained advisory | none | RustSec advisory database | Yes | |
| Deprecation notice | none | crates.io database dump | Yes | |
| Deprecated requirement | none | crates.io database dump | Yes |
Distribution4 releases, 30,659 downloads over 30 days5 not measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Releases | 4 | crates.io database dump | Yes | |
| Downloads over 30 days | 30659 | crates.io database dump | Yes | |
| Binary downloads over the last 3 releases | Not measured | Yes | ||
| Homebrew installs on request over 365 days | Not measured | Yes | ||
| Repology repository families | Not measured | Yes | ||
| Docker pull growth over 90 days | Not measured | Yes | ||
| Docker pulls at the growth baseline | Not measured | Yes | ||
| Every version a prerelease | false | crates.io database dump | Yes | |
| Every version 0.0.x | false | crates.io database dump | Yes | |
| Every version yanked | false | crates.io database dump | Yes | |
| Crate deleted from crates.io | false | crates.io database dump | Yes |
Repository2 stars, 1 forks, 2 contributors, 8 commits. Stars alone never qualify a project.
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| GitHub stars | 2 | GitHub repository | Yes — one part of the repository route | |
| GitHub forks | 1 | GitHub repository | Yes — one part of the repository route | |
| Contributors with a GitHub account | 2 | GitHub contributors | Yes — one part of the repository route | |
| Commits on the default branch | 8 | GitHub repository | No | |
| Repository's main crate left out of the entry | none | crates.io database dump | Yes |
Facts that never count
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| License | Apache-2.0 | crates.io database dump | No | |
| Runs on, from listed crates and their direct dependencies | GPU support unknown | crates.io database dump | No | |
| What shows each backend | no dependency or feature matches a backend row | crates.io database dump | No | |
| Native code, from listed crates and their dependencies | No native code found | crates.io database dump | No | |
| Python bindings, from direct dependencies of listed crates | No Python binding crate found | crates.io database dump | No | |
| Crates that depend on a listed crate directly | 5 | crates.io database dump | No |
Corrections and review history
No correction applies to this entry.
Judgments
A judgment enters only through an approved issue and a merged pull request.
No judgment applies to this entry.