alpino-tokenizer
Library Archived
Wrapper around the Alpino tokenizer for Dutch
- Downloads
- 20
- Dependents
- 0
- Stars
- 3
- Forks
- 0
- Contributors
- 1
- Commits
- 46
- Latest release
- 0.4.0 ·
Why Archived
GitHub reports the repository as archived.
Exact rules: Instant archive. The check on applied thresholds version 9.
Every rule outcome, as the check records it
instant-archive: repository is archived
About
This Rust crate provides a tokenizer based on finite state transducers. It is primarily designed to use the Alpino tokenizer for Dutch, but in principle, you could load a tokenizer for any language.
The transducer of the Alpino tokenizer can be downloaded. We will synchronize the transducer regularly as the tokenizer in Alpino is updated.
You can use the alpino-tokenizer crate to integrate the tokenizer in your Rust programs.
For convenience, an alpino-tokenize command-line utility is provided for tokenizing text on from the shell or in shell scripts.
Installing the alpino-tokenize command-line utility
cargo
The alpino-tokenize utility can be installed with
cargo:
$ cargo install alpino-tokenize
Nix
This repository is also a Nix flake. If you use a Nix version that
supports flakes, you can start a shell with alpino-tokenize as
follows:
$ nix shell github:danieldk/alpino-tokenizer
License
Copyright 2019-2020 Daniël de Kok
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Read the full README
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Compatibility
- Runs on
- GPU support unknown
What shows it
- no dependency or feature matches a backend row
- Native code
- Required
- Binds native code: alpino-tokenizer-sys
- Build uses a native toolchain: cc
- Python bindings
- No Python binding crate found
- License
- Apache-2.0
Compare with similar projects
Order: strongest tier first, then most downloads, then name, with unmeasured values last. A library counts downloads over 30 days. An app counts binary downloads per release. An app with none counts its crate downloads over 30 days. ML Rust runs this catalogue, so its closest project always gets a row.
| Project | Downloads | Dependents | Stars | Latest release | Runs on (GPU) | Native code | License |
|---|---|---|---|---|---|---|---|
|
Wrapper around the Alpino tokenizer for Dutch |
20 downloads over 30 days | 0 | 3 | 0.4.0 | GPU support unknown | Binds native code: alpino-tokenizer-sys. Build uses a native toolchain: cc | Apache-2.0 |
|
tiktoken-rs Featured
Library for encoding and decoding with the tiktoken library in Rust |
3,939,182 downloads over 30 days | 449 | 410 | 0.12.1 | GPU support unknown | No native code found | MIT |
|
sentencepiece Listed
Binding for the sentencepiece tokenizer |
30,249 downloads over 30 days | 18 | 24 | 0.14.0 | GPU support unknown | Binds native code: sentencepiece-sys. Build uses a native toolchain: cc, cmake, pkg-config | Apache-2.0 OR MIT |
|
splintr Listed
Tokenizer for Rust and Python: pure Rust by default, with 18 bundled vocabularies and four backends covering BPE, SentencePiece and WordPiece. Its output matches Hugging Face tokenizers id for id on 14 model families, including GPT-2, Qwen, Whisper, T5 and BERT. Python bindings |
8,204 downloads over 30 days | 7 | 63 | 0.22.2 | GPU support unknown | Binds native code: pcre2-sys (feature pcre2) | MIT |
|
kitoken Listed
Fast tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization |
2,949 downloads over 30 days | 3 | 62 | 0.11.1 | GPU support unknown | No native code found | BSD-2-Clause |
|
blingfire-rs Listed
Wrapper for the BlingFire tokenization library |
1,162 downloads over 30 days | 0 | 15 | 1.0.0 | GPU support unknown | Binds native code: blingfire-sys, blingfiretokdll. Build uses a native toolchain: cmake | MIT |
Crates3 crates
Measurements and sources
Each signal row shows the raw value, its source and the fetch date. The Sources list gathers every distinct source on this page. Movement needs more history.
Sources6 sources
- Description: the crates.io description
- crates.io database dump, fetched
- RustSec advisory database, fetched
- GitHub repository, fetched
- crates.io package, fetched
- GitHub contributors, fetched
Adoption0 active dependents, 0 new dependents in 365 days1 not measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Active dependents | 0 | crates.io database dump | Yes | |
| New dependents (365 days) | 0 | crates.io database dump | Yes | |
| Peak active dependents | 0 | crates.io database dump | Yes | |
| Active dependents as a share of the peak | 1 | crates.io database dump | Yes | |
| Binary downloads per release | Not measured | Yes |
MaintenanceLatest human commit: 3 years ago3 not measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Latest human commit | 2023-11-06 | GitHub repository | Yes | |
| Latest release | 2023-11-06 | crates.io database dump | Yes | |
| First release | 2019-10-11 | crates.io database dump | Yes | |
| Latest version | 0.4.0 | crates.io database dump | Yes | |
| Days the oldest outside issue waits for a reply | 0 | GitHub repository | Yes | |
| Maintainer replies over 365 days | 0 | GitHub repository | Yes | |
| Repository archived | true | GitHub repository | Yes | |
| Repository gone | false | GitHub repository | Yes | |
| Release commit or manifest in the repo | true | crates.io package | Yes | |
| Published docs | false | crates.io database dump | Yes | |
| Latest release fails cargo check | Not measured | Yes | ||
| Build check date | Not measured | Yes | ||
| Build check toolchain | Not measured | Yes |
SecurityUnpatched vulnerabilities: none
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Unpatched vulnerabilities past the grace window | none | RustSec advisory database | Yes | |
| RustSec unmaintained advisory | none | RustSec advisory database | Yes | |
| Deprecation notice | none | crates.io database dump | Yes | |
| Deprecated requirement | none | crates.io database dump | Yes |
Distribution4 releases, 20 downloads over 30 days5 not measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Releases | 4 | crates.io database dump | Yes | |
| Downloads over 30 days | 20 | crates.io database dump | Yes | |
| Binary downloads over the last 3 releases | Not measured | Yes | ||
| Homebrew installs on request over 365 days | Not measured | Yes | ||
| Repology repository families | Not measured | Yes | ||
| Docker pull growth over 90 days | Not measured | Yes | ||
| Docker pulls at the growth baseline | Not measured | Yes | ||
| Every version a prerelease | false | crates.io database dump | Yes | |
| Every version 0.0.x | false | crates.io database dump | Yes | |
| Every version yanked | false | crates.io database dump | Yes | |
| Crate deleted from crates.io | false | crates.io database dump | Yes |
Repository3 stars, 0 forks, 1 contributors, 46 commits. Stars alone never qualify a project.
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| GitHub stars | 3 | GitHub repository | Yes — one part of the repository route | |
| GitHub forks | 0 | GitHub repository | Yes — one part of the repository route | |
| Contributors with a GitHub account | 1 | GitHub contributors | Yes — one part of the repository route | |
| Commits on the default branch | 46 | GitHub repository | No | |
| Repository's main crate left out of the entry | none | crates.io database dump | Yes |
Facts that never count
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| License | Apache-2.0 | crates.io database dump | No | |
| Runs on, from listed crates and their direct dependencies | GPU support unknown | crates.io database dump | No | |
| What shows each backend | no dependency or feature matches a backend row | crates.io database dump | No | |
| Native code, from listed crates and their dependencies | Binds native code: alpino-tokenizer-sys. Build uses a native toolchain: cc | crates.io database dump | No | |
| Python bindings, from direct dependencies of listed crates | No Python binding crate found | crates.io database dump | No | |
| Crates that depend on a listed crate directly | 0 | crates.io database dump | No |
Corrections and review history
No correction applies to this entry.
Judgments
A judgment enters only through an approved issue and a merged pull request.
No judgment applies to this entry.