tiktoken-rs

use tiktoken_rs::o200k_base_singleton;

let bpe = o200k_base_singleton();
let tokens = bpe.encode_with_special_tokens(
  "This is a sentence   with spaces"
);
println!("Token count: {}", tokens.len());

Encoding text

CoreBPE::encode mirrors upstream tiktoken and returns a Result. Propagate or unwrap the result before using the tokens:

use tiktoken_rs::o200k_base;

let bpe = o200k_base().unwrap();
let allowed = bpe.special_tokens();
let (tokens, last_piece_token_len) = bpe.encode("hello <|endoftext|>", &allowed).unwrap();

The generic encode_as and count helpers also return Result. The crate requires Rust 1.85 or newer.

Counting max_tokens parameter for a chat completion request

use tiktoken_rs::{get_chat_completion_max_tokens, ChatCompletionRequestMessage};

let messages = vec![
    ChatCompletionRequestMessage {
        content: Some("You are a helpful assistant that only speaks French.".to_string()),
        role: "system".to_string(),
        ..Default::default()
    },
    ChatCompletionRequestMessage {
        content: Some("Hello, how are you?".to_string()),
        role: "user".to_string(),
        ..Default::default()
    },
    ChatCompletionRequestMessage {
        content: Some("Parlez-vous francais?".to_string()),
        role: "system".to_string(),
        ..Default::default()
    },
];
let max_tokens = get_chat_completion_max_tokens("o1-mini", &messages).unwrap();
println!("max_tokens: {}", max_tokens);

Counting max_tokens parameter for a chat completion request with async-openai

Enable the async-openai feature in your Cargo.toml file.

# #[cfg(feature = "async-openai")]
# {
use tiktoken_rs::async_openai::get_chat_completion_max_tokens;
use async_openai::types::chat::{
    ChatCompletionRequestMessage, ChatCompletionRequestSystemMessage,
    ChatCompletionRequestSystemMessageContent, ChatCompletionRequestUserMessage,
    ChatCompletionRequestUserMessageContent,
};

let messages = vec![
    ChatCompletionRequestMessage::System(ChatCompletionRequestSystemMessage {
        content: ChatCompletionRequestSystemMessageContent::Text(
            "You are a helpful assistant that only speaks French.".to_string(),
        ),
        name: None,
    }),
    ChatCompletionRequestMessage::User(ChatCompletionRequestUserMessage {
        content: ChatCompletionRequestUserMessageContent::Text(
            "Hello, how are you?".to_string(),
        ),
        name: None,
    }),
];
let max_tokens = get_chat_completion_max_tokens("o1-mini", &messages).unwrap();
println!("max_tokens: {}", max_tokens);
# }

tiktoken supports these encodings used by OpenAI models:

Context sizes

get_context_size uses the GPT-6 family default for any name beginning with gpt-6, including variants and snapshots. GPT-6 tokenizer lookup and token-budget helpers are unsupported because no encoding mapping is available.

See the examples in the repo for use cases. For more context on the different tokenizers, see the OpenAI Cookbook

Encountered any bugs?

If you encounter any bugs or have any suggestions for improvements, please open an issue on the repository.

Acknowledgements

Thanks @spolu for the original code, and .tiktoken files.

License

This project is licensed under the MIT License.