Text, embeddings and search

This crate aims to emulate https://github.com/google/sentencepiece Dart::DoubleArray struct and it's Normalizer. This crate is highly specialized and not intended for general use.

A rust implementation of some popular snowball stemming algorithms

Sentence segmentation library with wide language support optimized for speed and utility.

Fast byte-pair encoding implementation.

fastText pure Rust implementation

High-performance text analysis for full-text search

A blazingly fast and lightweight language detection library for Rust.

Inference engine for GLiNER models

High-performance Rust vector database core with HNSW indexing

Postgres extension for vector search (DiskANN), complements pgvector for performance and scale. Postgres OSS licensed.

Rust runtime reimplementation of SentencePiece model loading, normalization, encoding, and decoding.

Binding for the sentencepiece tokenizer

High performance tokenizers for Rust

Python bindings

A high-performance pure-Rust implementation of OpenAI's tiktoken BPE tokenizer

Ready-to-use NLP pipelines and language models

Library for constructing, combining, optimizing, and searching weighted finite-state transducers (FSTs).

Annoy-inspired Approximate Nearest Neighbors in Rust, based on LMDB and optimized for memory usage

Rust implementation of Snowball stemming algorithms for 33 languages

High-level bindings for Faiss, the vector similarity search engine

Fast approximate nearest neighbors

Rust bindings for OpenNMT/CTranslate2

A high-performance regex engine built from scratch with JIT compilation and SIMD acceleration

A fast, low-resource Natural Language Processing and Error Correction library.

Tokenizer for Rust and Python: pure Rust by default, with 18 bundled vocabularies and four backends covering BPE, SentencePiece and WordPiece. Its output matches Hugging Face tokenizers id for id on 14 model families, including GPT-2, Qwen, Whisper, T5 and BERT.

Python bindings

Yake (Yet Another Keyword Extractor) in Rust

Parse and convert numbers written in English, Danish, Dutch, Spanish, Portuguese, German, Italian or French into their digit representation.

A collection of Tantivy stemmer tokenizers

Optimized vector quantization for dense vectors

The fastest semantic text chunking library — up to 1TB/s chunking throughput

Core library for Memvid v2, a crash-safe, deterministic, single-file AI memory.

Reader and writer for common word embedding formats

Inverse Text Normalization (ITN) — convert spoken-form ASR output to written form

Rust bindings for UDPipe - a trainable pipeline for tokenization, tagging, lemmatization and dependency parsing of CoNLL-U files

Fast tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization

High-performance vector database engine written in Rust

Collection of algorithms for keyword extraction from text

Local first semantic and hybrid BM25 grep / search tool for use by AI and humans!

Hora Search Everywhere

A purpose-built database engine for AI agent memory

Fast token count estimation for LLMs at 96% accuracy without a full tokenizer

Wrapper for the BlingFire tokenization library

Vantage Point Tree search algorithm for fast nearest neighbour search in multi-dimensional metric spaces.

Postgres with GPUs for ML/AI apps.

Python bindings

Pure Rust tokenizer for GGUF models with llama.cpp compatibility (SentencePiece + BPE + WPM + UGM + RWKV)

Vector & lexical search engine library & multi-tenancy server

A fast and lightweight Rust library for splitting text into semantically meaningful chunks.

Suite of tools containing an in-memory vector datastore and AI proxy

Blazingly fast tokenizer - 50x faster tokenization, 10x smaller model files, 100% accurate drop-in replacement for HuggingFace

Semantic search and document parsing tools for the command line

Core async search engine library with WASM support

A Rust in-memory vector database

CPU-based PLAID implementation for multi-vector search using ndarray

Python bindings

AI-powered code intelligence with semantic search, knowledge graphs, and built-in MCP server. Transform your codebase into a queryable knowledge graph for AI assistants.

🔍 Tiny, full-text search engine for static websites built with Rust and Wasm

[H] HyperspaceDB is a high-performance, vector database. It features 1-bit quantization, async replication, and native support for hierarchical datasets (Lorentz, Poincaré ball & Hybrid models).

Rust wrappers for NGT nearest neighbor search.

NLP with Rust

Python bindings

Rust crate for entity parsing

Rust interface to word2vec.

Korvus is a search SDK that unifies the entire RAG pipeline in a single database query. Built on top of Postgres with bindings for Python, JavaScript, Rust and C.

Python bindings

Fast, lightweight and schema-less search backend.

A distributed search server.

YantrikDB database server — multi-tenant cognitive memory with wire protocol, HTTP gateway, replication, auto-failover, and at-rest encryption

Local-first, real-time, edge-to-cloud hybrid database for multi-modal workloads

and 1 related project

Graph-based Retrieval of Approximate Nearest Neighbors

LSH: Approximate Nearest Neighbor Search with Locality Sensitive Hashing.

Train/use word embeddings with subword units

Read and modify constituency trees.

Nested Object Inverted Search Engine

High-performance embedded vector database for Browser, Node, and Edge

Rust implementation of Sentence Bert (SBert)

All-in-one platform for search, recommendations, RAG, and analytics offered via API

Rust client for txtai

Seismic is designed for effective and efficient KNN retrieval over learned sparse embeddings.

Python bindings

An HDPSG inspired symbolic NLP library for Rust

Pure Rust 1-bit LLM inference engine for PrismML Bonsai models — umbrella crate

Persistent cognitive memory for AI agents and robots — Hebbian learning, knowledge graph, spatial recall. Zenoh/ROS2 native. Single binary, runs offline.

Python bindings

Cleora AI is a general-purpose open-source model for efficient, scalable learning of stable and inductive entity embeddings for heterogeneous relational data. Created by Synerise.com team.

Billion-scale vector database

Fast English word segmentation in Rust

A flexible, performant and reliable search database without the AI bullshit.

In-memory vector store with efficient read and write performance for semantic caching and retrieval system. Redis for Semantic Caching.

Rust-based Natural Language Toolkit

Python bindings

High performance embedded vector database

Ontology of Snips NLU

Sudachi in Rust 🦀 and new generation of SudachiPy

A library that provides the ability to convert tokens into embeddings, as well as to encode their positions

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

A browser-optimized vector database

Hamming Weight Tree from the paper "Online Nearest Neighbor Search in Hamming Space"

A fast, searchable, knowledge engine using various machine learning models to aggregate based on importance, association and relevance

Natural Language Processing for Rust

A tiny embedding database in pure Rust.

A full-text search engine in rust

🕸️🦀 A WASM vector similarity search written in Rust

pure rust implemention of word2vec