rust-gems

• geo_filters: probabilistic data structures that solve the Distinct Count Problem using geometric filters.
• bpe: fast, correct, and novel algorithms for the Byte Pair Encoding Algorithm which are particularly useful for chunking of documents.
• bpe-openai: Fast tokenizers for OpenAI token sets based on the bpe crate.
• commutative_hasher: hash function that makes it possible to hash large blobs in parallel chunks.
• consistent-choose-k: constant time consistent hashing algorithms with support for replication and bounded load.
• hash-sorted-map: a hash map whose groups are ordered by hash prefix, enabling efficient sorted-order iteration and linear-time merging.
• sparse-ngrams: fast sparse n-gram extraction from byte slices. Selects variable-length n-grams (2–8 bytes) deterministically using bigram frequency priorities, suitable for substring search indexes.
• string-offsets: converts string positions between bytes, chars, UTF-16 code units, and line numbers. Useful when sending string indices across language boundaries.
• casefold: a fast Unicode simple case-folding library backed by a very compact (~1.7 KB) paged-bitmap + run-length table. Folds whole strings at multiple GiB/s via a decode-free simple_fold that rewrites UTF-8 with little-endian byte arithmetic, beating a HashMap fold table by several × at ~10× less memory.

Background

Rust Algorithms is under active development and maintained by GitHub staff AND THE COMMUNITY. See CODEOWNERS for more details.

We will do our best to respond to support, feature requests, and community questions in a timely manner. For more details see support and contribution guidelines.

Requirements

Requires a working installation of Rust through download | through Homebrew

License

This project is licensed under the terms of the MIT open source license. Please refer to MIT for the full terms.