rust-tiktoken

The two crates live in one workspace and are versioned in lockstep — every release bumps and publishes both at the same version number.

Highlights

• Hand-written pre-tokenizer for ASCII and CJK — letters, digits, punctuation, contractions, Han/kana/hangul runs and fullwidth forms all resolve without the regex engine, which remains the arbiter (property-tested equivalence) and the fallback for rare shapes.
• 17 encodings · 116 models · 11 providers — OpenAI (GPT-4/4o/4.1/4.5, GPT-5.x, o1/o3/o4-mini, gpt-oss), Llama 3/4, DeepSeek V3/V4, Qwen, Mistral, Kimi K2/K3, GLM-4/5, MiniMax M2; plus USD cost estimation (Anthropic & Google included for pricing).
• Lean & portable — all 17 vocabularies embedded in 5.1 MB and opt-out per vocabulary (a cl100k-only build carries 373 KB), key-size-layered vocabulary with whole-piece memoisation, hybrid linear/heap BPE merge, optional rayon parallelism, allocation-free count(), pure Rust with zero C dependencies, and a self-contained wasm build.

Full API, supported-model tables, and benchmarks live in the per-crate READMEs: tiktoken/ · tiktoken-wasm/.

Quick start

Rust

[dependencies]
tiktoken = "4"

// by encoding name
let enc = tiktoken::get_encoding("cl100k_base").unwrap();
let tokens = enc.encode("hello world");
assert_eq!(enc.decode_to_string(&tokens).unwrap(), "hello world");

// count without allocating a token vector
let n = enc.count("The quick brown fox.");

// or resolve by model name
let enc = tiktoken::encoding_for_model("gpt-4o").unwrap();

WebAssembly (browser / Node.js)

npm install @goliapkg/tiktoken-wasm

import init, { getEncoding } from '@goliapkg/tiktoken-wasm'
await init()
const enc = getEncoding('o200k_base')
const tokens = enc.encode('hello world')

Performance

On an Apple M4 Mac mini, encode is 5–49x faster than tiktoken-rs and 5–29x faster than Python tiktoken: 29–49x on ASCII, 15–17x on Chinese and Japanese prose, 5x even on an adversarial no-repeat CJK corpus. In the browser (wasm) it is 2–4x faster than gpt-tokenizer. Full tables and methodology: tiktoken/README.md#performance.

Build

cargo test -p tiktoken
cargo fmt --all --check
cargo clippy --workspace --lib -- -D warnings

# WASM (requires wasm-pack: cargo install wasm-pack)
cd tiktoken-wasm
wasm-pack build --target web --release --scope goliapkg

Release

tiktoken and tiktoken-wasm are versioned in lockstep and released together via git-flow (no PRs):

# release gate: the full property sweep (~35 minutes; CI runs a smoke sample)
PROPTEST_CASES=25000 cargo test --workspace --all-targets

git flow release start X.Y.Z
# bump versions to X.Y.Z: tiktoken/Cargo.toml, tiktoken-wasm/Cargo.toml
# (and its tiktoken path-dep); finalize both CHANGELOGs
git flow release finish X.Y.Z                       # merge → master, tag vX.Y.Z, back-merge develop
git tag -a tiktoken-wasm-vX.Y.Z vX.Y.Z^{commit} -m "tiktoken-wasm X.Y.Z"
git push origin master develop vX.Y.Z tiktoken-wasm-vX.Y.Z
# tag `v*` publishes the tiktoken crate; `tiktoken-wasm-v*` publishes the wasm crate + npm

The 23 pre-tokenizer and 12 round-trip property tests pin the hand-written scanners against the regex engine. PROPTEST_CASES sets how much new ground each run covers; the seeds in proptest-regressions/ replay before any random case at every setting, so past failures are always re-checked.

License

Licensed under either of MIT or Apache-2.0, at your option.