Text and tokenizers

This crate aims to emulate https://github.com/google/sentencepiece Dart::DoubleArray struct and it's Normalizer. This crate is highly specialized and not intended for general use.

A rust implementation of some popular snowball stemming algorithms

Sentence segmentation library with wide language support optimized for speed and utility.

Fast byte-pair encoding implementation.

Pure Rust PDF library for AI/RAG: structure-aware chunking, no ML, no C deps.

High-performance document intelligence library for Rust. Extract text, metadata, and structured data from PDFs, Office documents, images, and 91+ formats and 248 programming languages via tree-sitter code intelligence with async/sync APIs.

A blazingly fast and lightweight language detection library for Rust.

Inference engine for GLiNER models

Rust runtime reimplementation of SentencePiece model loading, normalization, encoding, and decoding.

Binding for the sentencepiece tokenizer

High performance tokenizers for Rust

Python bindings

A high-performance pure-Rust implementation of OpenAI's tiktoken BPE tokenizer

Ready-to-use NLP pipelines and language models

Rust implementation of Snowball stemming algorithms for 33 languages

A high-performance regex engine built from scratch with JIT compilation and SIMD acceleration

A fast, low-resource Natural Language Processing and Error Correction library.

Tokenizer for Rust and Python: pure Rust by default, with 18 bundled vocabularies and four backends covering BPE, SentencePiece and WordPiece. Its output matches Hugging Face tokenizers id for id on 14 model families, including GPT-2, Qwen, Whisper, T5 and BERT.

Python bindings

Token-Oriented Object Notation – a token-efficient JSON alternative for LLM prompts

Yake (Yet Another Keyword Extractor) in Rust

EdgeParse core library — PDF parsing and structured data extraction

Parse and convert numbers written in English, Danish, Dutch, Spanish, Portuguese, German, Italian or French into their digit representation.

A collection of Tantivy stemmer tokenizers

The fastest semantic text chunking library — up to 1TB/s chunking throughput

Rust bindings for UDPipe - a trainable pipeline for tokenization, tagging, lemmatization and dependency parsing of CoNLL-U files

Fast tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization

A Serde-compatible TOON (Token-Oriented Object Notation) serialization library

Collection of algorithms for keyword extraction from text

Extractous provides a fast and efficient way to extract content from all kind of file formats including PDF, Word, Excel CSV, Email etc... Internally it uses a natively compiled Apache Tika for formats are not supported natively by the Rust core

Fast token count estimation for LLMs at 96% accuracy without a full tokenizer

Wrapper for the BlingFire tokenization library

Pure Rust tokenizer for GGUF models with llama.cpp compatibility (SentencePiece + BPE + WPM + UGM + RWKV)

A fast and lightweight Rust library for splitting text into semantically meaningful chunks.

Blazingly fast tokenizer - 50x faster tokenization, 10x smaller model files, 100% accurate drop-in replacement for HuggingFace

NLP with Rust

Python bindings

Rust crate for entity parsing

Read and modify constituency trees.

Rust client for txtai

An HDPSG inspired symbolic NLP library for Rust

Vision infrastructure to turn complex documents into LLM-ready data

Fast English word segmentation in Rust

Rust-based Natural Language Toolkit

Python bindings

Ontology of Snips NLU

Sudachi in Rust 🦀 and new generation of SudachiPy

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

A fast, searchable, knowledge engine using various machine learning models to aggregate based on importance, association and relevance

Natural Language Processing for Rust