tokie

Quick start

Python

import tokie

tokenizer = tokie.Tokenizer.from_pretrained("google-bert/bert-base-uncased")

ids = tokenizer.encode("Hello, world!")
print(ids.tolist())

plus = tokenizer.encode_plus("Hello, world!")
print(plus.tolist())
print(tokenizer.encode_plus("Hello", pair="world", truncate=True, pad=True, max_len=8).tolist())

print(tokenizer.decode(plus.tolist(), skip=True))

flat, ends = tokenizer.batch(["Hello!", "World"])
print(tokenizer.count_tokens("Hello, world!"))
print(tokenizer.count_tokens_batch(["Hello!", "World"]))

encode returns a uint32 array and leaves the template off. encode_plus writes the single-sequence template, so BERT uncased is [CLS], the ids, [SEP]. pair uses the pair template. truncate and pad share max_len, and that length counts the template ids. batch returns one id array and ends[i], the id count after text i. count_tokens returns the length encode would append. count_tokens_batch returns one length per text. Both leave the template off.

Rust

use tokie::{Plus, Specials, Tokenizer};

let mut tokenizer = Tokenizer::pretrained("google-bert/bert-base-uncased")?;
let mut ids = Vec::new();
tokenizer.encode("Hello, world!", &mut ids)?;

ids.clear();
tokenizer.encode_plus("Hello, world!", &mut ids)?;
tokenizer.encode_plus(Plus::new("Hello").pair("world").truncate(8).pad(8), &mut ids)?;

let mut text = String::new();
tokenizer.decode(&ids, &mut text, Specials::Skip)?;

let n = tokenizer.count_tokens("Hello, world!")?;
let lengths = tokenizer.count_tokens_batch(&["Hello!", "World"])?;

encode and encode_plus append into the Vec the caller owns. Clear it to start a new sequence. count_tokens and count_tokens_batch write ids into a buffer on the tokenizer and return the lengths. They leave the template off.

Examples

Batch

tokenizer.threads(4)
flat, ends = tokenizer.batch(["Short text", "A much longer piece of text"])

tokenizer.threads(4);
let lines = ["Short text", "A much longer piece of text"];
let mut ids = Vec::new();
let mut ends = Vec::new();
tokenizer.batch(&lines, &mut ids, &mut ends)?;

Text i is ids[prev..ends[i]]. ends[i] is ids.len() after that text.

Vocabulary

tokenizer.id("[SEP]")
tokenizer.token(101)

tokenizer.id("[SEP]");
tokenizer.token(101);

Save and load a .tokie file

tokenizer.save("model.tokie")
again = tokie.Tokenizer.open("model.tokie")

tokenizer.save("model.tokie")?;
let mut again = Tokenizer::open("model.tokie")?;

from_pretrained in Python, and pretrained in Rust, reads the repo's .tokie file and then tokenizer.json.

Benchmarks

All benchmarks run on an Apple M3 with tokie 0.1.3. The ids match Hugging Face tokenizers.

BPE encoding (GPT-2, Llama, Qwen, ModernBERT)

For tiktoken-style BPE, tokie uses a backtracking encoder on an Aho-Corasick automaton. It takes a greedy longest match in O(n), and it backtracks only when two adjacent tokens are not a valid pair. It does not merge one byte pair at a time. Pretokenization is the SIMD mask scanner in pretokie. That scanner classifies 64-byte blocks into piece-boundary bitmasks and runs above 1 GB/s per core. A repeated piece hits a per-worker cache instead of running BPE again. Work stealing splits the text so every core has a chunk. On one string this is 39 to 154 times faster than Hugging Face tokenizers.

WordPiece (BERT, MiniLM, BGE, GTE)

WordPiece searches a vocabulary trie for the longest prefix. tokie stores that trie as a Double-Array trie and looks it up in O(n), with a BERT pretokenizer. On BERT this is 25 to 71 times faster than Hugging Face tokenizers, and the ids match.

SentencePiece BPE and Unigram (Gemma, XLM-R, T5)

SentencePiece BPE merges in an order that is not topological. tokie uses a radix heap. Insert and pop are O(1) amortized because a later BPE rank is never smaller. On Gemma 3 this is 2 to 3 times faster than Hugging Face tokenizers.

Unigram models such as T5, XLM-RoBERTa, and ALBERT run Viterbi on each metaspace-delimited word and store the frequent units. Encoding one string is about 4 times faster than whole-string Viterbi. Batch encoding is about 3 times faster. The ids match. A vocabulary with a metaspace inside a token uses whole-string Viterbi.

Python benchmarks

All results on Apple M3, single-string encode, median of 10 runs.

tokie vs Hugging Face tokenizers

tokie vs tiktoken (OpenAI models)

The ids match on every model in these tables. Batch encoding is 17 to 36 times faster than Hugging Face tokenizers. encode_batch_flat returns one contiguous NumPy array.

Tokenizer loading

Loading tokenizer.json parses the JSON, builds the vocabulary, and, for BPE, builds the Aho-Corasick automaton. tiktoken parses its BPE data and compiles its regexes on every load. A .tkz file stores the Double-Array Aho-Corasick state, the normalized vocabulary, the encoder configuration, and, since v13, the added and special tokens. Loading that file is deserialization. from_pretrained reads the hub disk cache before the network and writes the compiled .tkz next to the snapshot, so a warm load does not use the network. A warm load is 13 to 44 times faster than Hugging Face tokenizers. BERT takes 2.8 ms. GPT-2 takes 6.2 ms.

Verified tokenizers

A passing tokenizer is tested against the Hugging Face tokenizer on 1 MB of enwik8, about 300,000 tokens, in CI. Pass means every token matches. Not supported means tokie 0.2 refuses the file.

The full matrix is in docs/TOKENIZERS.md.

Full accuracy table, 75 models

69 pass. 6 are not supported, out of 75 listed. The passing tokenizers' ids match Hugging Face tokenizers.

Why tokie?

When I started building Chonkie, tokenization took longer than chunking.

Each pretokenization pattern has its own parser: GPT-2, cl100k, o200k, and BERT. The parser uses that pattern's character classes and does not run a general regex engine. Since 0.1.0 those parsers are a SIMD mask scanner. They classify 64-byte blocks into character-class bitmasks and take piece boundaries from the masks. Pretokenization passes 1 GB/s per core, more than 20 times the reference regexes.

No single library loaded every Hugging Face tokenizer. I tried AutoTikTokenizer and expected tiktoken's BPE to cover them. It does not. Tiktoken-style models need backtracking BPE. A merge order that is not topological needs a heap. SentencePiece needs a radix heap. WordPiece and Unigram each need their own encoder.

Tokenization runs in parallel when the split keeps token boundaries. chunk splits the text on those boundaries. Each core encodes one chunk, and the ids are concatenated. On 8 cores that is about 5 times faster.

.tkz skips rebuilding those structures at load time. tokenizer.json is parsed and validated, and the Aho-Corasick automaton is built. That build is slow for a large vocabulary. .tkz stores the automaton, the vocabulary, the configuration, and the added tokens as a flat binary. Loading deserializes that file. With the hub cache and a compiled file on disk, a warm BERT load is 2.8 ms and a warm GPT-2 load is 6.2 ms. Hugging Face tokenizers takes 120 to 310 ms on those two models.

Acknowledgements

tokie uses ideas from Hugging Face tokenizers, tiktoken, and GitHub's rust-gems for backtracking BPE via Aho-Corasick. It also uses chunk for SIMD text splitting, and gigatoken. Gigatoken's piece cache, mask-scanner pretokenization, and benchmarks shaped the 0.1.0 hot path.

Citation

If you use tokie in your research, please cite it as follows:

@software{tokie2025,
  author = {Minhas, Bhavnick},
  title = {tokie: Fast, correct tokenizer library for every Hugging Face model},
  year = {2025},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/chonkie-inc/tokie}},
}