tokenx-rs

Rust port of tokenx by Johann Schopplich

Fast token count estimation for LLMs at 96% accuracy without a full tokenizer.

Why?

• No vocabulary files: Full tokenizers like tiktoken require 2-4MB of BPE vocabulary data
• Zero dependencies: No regex, no allocations — just a single-pass character scanner
• Fast: ~8x faster than the original Node.js implementation on Latin text
• Accurate enough: 96% accuracy is sufficient for token budget estimation and streaming display
• Universal: Works across all LLM providers (OpenAI, Anthropic, Google, etc.)

Installation

[dependencies]
tokenx-rs = "0.1"

Usage

use tokenx_rs::estimate_token_count;

let tokens = estimate_token_count("Hello, world!");
println!("Estimated tokens: {}", tokens);

Check token limits

use tokenx_rs::is_within_token_limit;

if is_within_token_limit(prompt, 4096) {
    // safe to send
}

Split text into chunks

use tokenx_rs::split_by_tokens;

let chunks = split_by_tokens(long_text, 1000);

How it works

The estimator makes a single pass over the input string, classifying each character and grouping runs of similar characters into segments. Each segment is scored by type: