ML Rust catalogue
Menu

whatlang-rs

Library Featured

Fast and lightweight language identification library for Rust.

Checked · Sources

Downloads
882,019
Dependents
74
Stars
1,087
Forks
122
Contributors
21
Commits
585
Latest release
0.18.0 ·

About

From the project README ·

Whatlang - rust library for natural language detection

Natural language detection for Rust with focus on simplicity and performance.

Try online demo.

Stand With Ukraine

Content

  • Features
  • Get started
  • Who uses Whatlang?
  • Documentation
  • Supported languages
  • Feature toggles
  • How does it work?
  • Make tasks
  • Comparison with alternatives
  • Ports and clones
  • Donations
  • Derivation
  • License
  • Contributors

Features

  • Supports 70 languages
  • 100% written in Rust
  • Lightweight, fast and simple
  • Recognizes not only a language, but also a script (Latin, Cyrillic, etc)
  • Provides reliability information

Get started

Example:

use whatlang::{detect, Lang, Script};

fn main() {
    let text = "Ĉu vi ne volas eklerni Esperanton? Bonvolu! Estas unu de la plej bonaj aferoj!";

    let info = detect(text).unwrap();
    assert_eq!(info.lang(), Lang::Epo);
    assert_eq!(info.script(), Script::Latin);
    assert_eq!(info.confidence(), 1.0);
    assert!(info.is_reliable());
}

For more details (e.g. how to blacklist some languages) please check the documentation.

Who uses Whatlang?

Whatlang is used within the following big projects as direct or indirect dependency for language recognition. You're gonna be in a great company using Whatlang:

Read the full README
  • Sonic - fast, lightweight and schema-less search backend in Rust.
  • Meilisearch - an open-source, easy-to-use, blazingly fast, and hyper-relevant search engine built in Rust.

Feature toggles

FeatureDescription
enum-mapLang and Script implement Enum trait from enum-map
arbitrarySupport Arbitrary
serdeImplements Serialize and Deserialize for Lang and Script
devEnables whatlang::dev module which provides some internal API.
It exists for profiling purposes and normal users are discouraged to to rely on this API.

How does it work?

How does the language recognition work?

The algorithm is based on the trigram language models, which is a particular case of n-grams. To understand the idea, please check the original whitepaper Cavnar and Trenkle '94: N-Gram-Based Text Categorization'.

How is is_reliable calculated?

It is based on the following factors:

  • How many unique trigrams are in the given text
  • How big is the difference between the first and the second(not returned) detected languages? This metric is called rate in the code base.

Therefore, it can be presented as 2d space with threshold functions, that splits it into "Reliable" and "Not reliable" areas. This function is a hyperbola and it looks like the following one:

Language recognition whatlang rust

For more details, please check a blog article Introduction to Rust Whatlang Library and Natural Language Identification Algorithms.

Make tasks

  • make bench - run performance benchmarks
  • make doc - generate and open doc
  • make test - run tests
  • make watch - watch changes and run tests

Comparison with alternatives

WhatlangCLD2CLD3
Implementation languageRustC++C++
Languages7083107
Algorithmtrigramsquadgramsneural network
Supported EncodingUTF-8UTF-8?
HTML supportnoyes?

Ports and clones

Donations

You can support the project by donating NEAR tokens.

Our NEAR wallet address is whatlang.near

Derivation

Whatlang is a derivative work from Franc (JavaScript, MIT) by Titus Wormer.

License

MIT © Sergey Potapov

Contributors

Compatibility

How each fact is measured

Runs on
GPU support unknown
What shows it
  • no dependency or feature matches a backend row
Native code
None found
  • No native code found
Python bindings
No Python binding crate found
License
MIT

Compare with similar projects

Order: strongest tier first, then most downloads, then name, with unmeasured values last. A library counts downloads over 30 days. An app counts binary downloads per release. An app with none counts its crate downloads over 30 days. ML Rust runs this catalogue, so its closest project always gets a row.

ProjectDownloadsDependentsStarsLatest releaseRuns on (GPU)Native codeLicense
whatlang-rs This page Featured

Fast and lightweight language identification library for Rust.

 74 1,087 0.18.0 MIT
tiktoken-rs Featured

Library for encoding and decoding with the tiktoken library in Rust

 449 410 0.12.1 MIT
lingua-rs Featured

An accurate natural language detection library, suitable for short text and mixed-language text

Python bindings

 22 1,152 1.8.0 Apache-2.0
sentencex Listed

Sentence segmentation library with wide language support optimized for speed and utility.

 1 145 0.1.34 MIT
whichlang Listed

A blazingly fast and lightweight language detection library for Rust.

 11 464 0.1.1 MIT
splintr Listed

Tokenizer for Rust and Python: pure Rust by default, with 18 bundled vocabularies and four backends covering BPE, SentencePiece and WordPiece. Its output matches Hugging Face tokenizers id for id on 14 model families, including GPT-2, Qwen, Whisper, T5 and BERT.

Python bindings

 7 63 0.22.2 MIT
semchunk-rs Listed Dormant

A fast and lightweight Rust library for splitting text into semantically meaningful chunks.

 3 4 0.1.1 MIT

Compare these

All projects in Text, embeddings and search

Why FeaturedIt passes every Featured gate.

What it meets

  • It has 52 active dependents and 45 new dependents in 365 days. Featured needs 11 active dependents or 9 new dependents in 365 days.
  • Active dependents are at 100% of their peak. Featured needs at least 50%.
  • Docs are present. The release commit or a manifest ties the crate to the repository.
  • No vulnerability advisory stays unpatched past the grace window. At least one version is neither a prerelease nor 0.0.x.
  • The last activity is on 24 Dec 2025, 320 days ago. The rules expect an update within 560 days. Dormancy rule

Exact rules: Featured library. The check on applied thresholds version 9.

Every rule outcome, as the check records it
  • expected-interval: version 0.18.0 base 140, doubled for 882019 downloads per month, doubled for 51.5 active users: E = 560 days
  • dormancy: last activity 2025-12-24, 320 days ago, E = 560: active
  • featured: featured: active_users 51.5, new_adopters 44.5, former_glory 1

Measurements and sources

Each signal row shows the raw value, its source and the fetch date. The Sources list gathers every distinct source on this page. Movement needs more history.

Sources6 sources
Adoption52 active dependents, 45 new dependents in 365 days1 not measured
SignalValueSourceFetchedCounts
Active dependents51.5crates.io database dumpYes
New dependents (365 days)44.5crates.io database dumpYes
Peak active dependents50.5crates.io database dumpYes
Active dependents as a share of the peak1crates.io database dumpYes
Binary downloads per releaseNot measuredYes
MaintenanceLatest human commit: 10 months ago3 not measured
SignalValueSourceFetchedCounts
Latest human commit2025-12-24GitHub repositoryYes
Latest release2025-10-16crates.io database dumpYes
First release2016-12-15crates.io database dumpYes
Latest version0.18.0crates.io database dumpYes
Days the oldest outside issue waits for a reply539GitHub repositoryYes
Maintainer replies over 365 days2GitHub repositoryYes
Repository archivedfalseGitHub repositoryYes
Repository gonefalseGitHub repositoryYes
Release commit or manifest in the repotruecrates.io packageYes
Published docstruecrates.io database dumpYes
Latest release fails cargo checkNot measuredYes
Build check dateNot measuredYes
Build check toolchainNot measuredYes
SecurityUnpatched vulnerabilities: none
SignalValueSourceFetchedCounts
Unpatched vulnerabilities past the grace windownoneRustSec advisory databaseYes
RustSec unmaintained advisorynoneRustSec advisory databaseYes
Deprecation noticenonecrates.io database dumpYes
Deprecated requirementnonecrates.io database dumpYes
Distribution35 releases, 882,019 downloads over 30 days5 not measured
SignalValueSourceFetchedCounts
Releases35crates.io database dumpYes
Downloads over 30 days882019crates.io database dumpYes
Binary downloads over the last 3 releasesNot measuredYes
Homebrew installs on request over 365 daysNot measuredYes
Repology repository familiesNot measuredYes
Docker pull growth over 90 daysNot measuredYes
Docker pulls at the growth baselineNot measuredYes
Every version a prereleasefalsecrates.io database dumpYes
Every version 0.0.xfalsecrates.io database dumpYes
Every version yankedfalsecrates.io database dumpYes
Crate deleted from crates.iofalsecrates.io database dumpYes
Repository1,087 stars, 122 forks, 21 contributors, 585 commits. Stars alone never qualify a project.
SignalValueSourceFetchedCounts
GitHub stars1087GitHub repositoryYes — one part of the repository route
GitHub forks122GitHub repositoryYes — one part of the repository route
Contributors with a GitHub account21GitHub contributorsYes — one part of the repository route
Commits on the default branch585GitHub repositoryNo
Repository's main crate left out of the entrynonecrates.io database dumpYes
Facts that never count
SignalValueSourceFetchedCounts
LicenseMITcrates.io database dumpNo
Runs on, from listed crates and their direct dependenciesGPU support unknowncrates.io database dumpNo
What shows each backendno dependency or feature matches a backend rowcrates.io database dumpNo
Native code, from listed crates and their dependenciesNo native code foundcrates.io database dumpNo
Python bindings, from direct dependencies of listed cratesNo Python binding crate foundcrates.io database dumpNo
Crates that depend on a listed crate directly74crates.io database dumpNo

Corrections and review history

Report a correction

No correction applies to this entry.

Judgments

A judgment enters only through an approved issue and a merged pull request.

No judgment applies to this entry.