ungoliant
Library Listed Dormant
The pipeline for the OSCAR corpus.
- Downloads
- 1
- Dependents
- 0
- Stars
- 178
- Forks
- 17
- Contributors
- 5
- Commits
- 419
- Latest release
- 2.0.0 ·
About
🕷️ Ungoliant is a high-performance pipeline that provides tools to build corpus generation pipelines from CommonCrawl. 🕷️
It currently is the generation pipeline for OSCAR corpus, from CommonCrawl. Ungoliant is a replacement of goclassy.
Installation
Installing/Compiling the binary
- Via
cargo:cargo install ungoliant - Via
git:cargo install --git https://github.com/oscar-corpus/ungoliant
Ungoliant needs numerous dependencies that should be compiled when installing. However cmake / gcc can be needed as the project uses fasttext-rs.
KenLM feature
The KenLM feature is optional because it relies on unsafe code that can break if the supplied model files are not correct.
To enable it, install KenLM requirements:
apt install -y libboost-all-dev libeigen3-dev
and use cargo install ungoliant --features kenlm or cargo b --features kenlm if you're building from source.
Getting a language identification file (for fastText):
By default, ungoliant expects the lid.176.bin model by meta.
Use curl https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin -o lid.176.bin to get it.
However, you can use the model you want: just point to its path using ungoliant download --lid-path <path to lid>.
Read the full README
Other options include:
- NLLB model (https://huggingface.co/facebook/fasttext-language-identification)
- OpenLID model (https://github.com/laurieburchell/open-lid-dataset)
Usage
The usual way of generating corpora is:
- Fetch the
wet.paths.gzfile from the last CommonCrawl dump and decompress it. - Download the files using the
downloadcommand. - Generate the corpus using the
pipelinecommand (it may take some time). - Head on to oscar-tools for the packaging steps
You can find more information on each command's --help.
ungoliant 2
corpus generation tool.
USAGE:
ungoliant <SUBCOMMAND>
FLAGS:
-h, --help Prints help information
-V, --version Prints version information
SUBCOMMANDS:
download Download a CommonCrawl release
help Prints this message or the help of the given subcommand(s)
pipeline Run pipeline
rebuild Rebuild the corpus for a given language.
Documentation
Ungoliant is not yet on docs.rs: use cargo doc --bins --open to open the documentation.
Head on to OSCAR Documentation for more info about the project.
Compatibility
- Runs on
- GPU support unknown
What shows it
- no dependency or feature matches a backend row
- Native code
- Required
- Binds native code: zstd-sys
- Python bindings
- No Python binding crate found
- License
- Apache-2.0
Compare with similar projects
Order: strongest tier first, then most downloads, then name, with unmeasured values last. A library counts downloads over 30 days. An app counts binary downloads per release. An app with none counts its crate downloads over 30 days. ML Rust runs this catalogue, so its closest project always gets a row.
| Project | Downloads | Dependents | Stars | Latest release | Runs on (GPU) | Native code | License |
|---|---|---|---|---|---|---|---|
|
The pipeline for the OSCAR corpus. |
1 downloads over 30 days | 0 | 178 | 2.0.0 | GPU support unknown | Binds native code: zstd-sys | Apache-2.0 |
|
datafusion Featured
DataFusion is an in-memory query engine that uses Apache Arrow as the memory model |
2,992,524 downloads over 30 days | 484 | 9,578 | 55.2.0 | GPU support unknown | Binds native code: libmimalloc-sys, liblzma-sys (feature compression), zstd-sys (feature compression) | Apache-2.0 |
|
lance Featured
A columnar data format that is 100x faster than Parquet for random access. |
915,526 downloads over 30 days | 48 | 7,268 | 14.0.2 | GPU support unknown | Binds native code: ring (feature credential-vendor-gcp), lz4-sys (feature lz4), zstd-sys (feature zstd). Build uses a native toolchain: cc | Apache-2.0 |
|
hdf5-rust Featured
Thread-safe Rust bindings for the HDF5 library. |
253,180 downloads over 30 days | 70 | 85 | 0.15.1 | GPU support unknown | Binds native code: hdf5-metno-sys, blosc-src (feature blosc), lzf-sys (feature lzf), mpi-sys (feature mpio), zfp-sys (feature zfp) | Apache-2.0 OR MIT |
|
zarrs Featured
A library for the Zarr storage format for multidimensional arrays and metadata |
75,052 downloads over 30 days | 51 | 285 | 0.23.14 | GPU support unknown | Binds native code: blosc-src (feature blosc), gdeflate-sys (feature gdeflate), zfp-sys (feature zfp), zstd-sys (feature zstd) | Apache-2.0 OR MIT |
|
vortex Listed
An extensible, state-of-the-art framework for columnar compression, and the fastest FOSS columnar file format. Formerly at @spiraldb, now an Incubation Stage project at LFAI&Data, part of the Linux Foundation. |
98,129 downloads over 30 days | 32 | 3,310 | 0.93.0 | GPU support unknown | Binds native code: zstd-sys | Apache-2.0 |
|
Model conversion and quantization with no Python: reads SafeTensors, GGUF, TCF and PyTorch state dicts, writes SafeTensors, GGUF and TCF, and quantizes to Q2_K through Q8_0 and IQ4 formats, including Q4_K_M mixes. GPTQ, AWQ, NF4, FP8, EXL2 and ONNX input are not implemented. Python bindings |
23 downloads over 30 days | Not applicable | 0 | 0.2.1 | GPU support unknown | No native code found | Apache-2.0 |
Why ListedNo archive, New or Featured rule places it, so it is Listed.
What it lacks for Featured
- It has 0 active dependents and 0 new dependents in 365 days. Featured needs 11 active dependents or 9 new dependents in 365 days. The repository route reads 178 stars, 17 forks and 5 contributors. It needs 4,539 stars, 434 forks and 79 contributors together.
- The project is dormant and not finished. Dormancy rule
What else the check found
- The last activity is on 9 Nov 2023, 1096 days ago. That reaches the 365 days the rules expect, so the project is dormant. Dormancy rule
- The oldest outside issue waits 1861 days for a reply, past the limit of 270. No maintainer replied in 365 days. Rot signals rule
- The latest release requires the deprecated dependency structopt. RUSTSEC-2022-0104 Rot signals rule
- The project is dormant with 2 rot signals, which reaches the limit of 2. This is dead check 1 of 2, so the entry is not yet archived. Dead rule
Not measured: Latest release fails cargo check.
Exact rules: Listed, Featured library. The check on applied thresholds version 9.
Every rule outcome, as the check records it
expected-interval: version 2.0.0 base 200, raised to the 365-day floor: E = 365 daysdormancy: last activity 2023-11-09, 1096 days ago, E = 365: dormantrot: oldest unanswered issue 1861 days (limit 270), no maintainer reply in 365 daysrot: deprecated requirement `structopt (RUSTSEC-2022-0104, https://rustsec.org/advisories/RUSTSEC-2022-0104.html)`dead: dormant with 2 rot signals (limit 2): dead on check 1 of 2, not yet archivedfeatured: not featured: active_users 0 < 11 and new_adopters 0 < 9 and repository route stars 178 (needs 4539), forks 17 (needs 434), contributors 5 (needs 79); dormant and not stablelisted: listed: no archive, new or featured rule applies
Measurements and sources
Each signal row shows the raw value, its source and the fetch date. The Sources list gathers every distinct source on this page. Movement needs more history.
Sources6 sources
- Description: the crates.io description
- crates.io database dump, fetched
- RustSec advisory database, fetched
- GitHub repository, fetched
- crates.io package, fetched
- GitHub contributors, fetched
Adoption0 active dependents, 0 new dependents in 365 days1 not measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Active dependents | 0 | crates.io database dump | Yes | |
| New dependents (365 days) | 0 | crates.io database dump | Yes | |
| Peak active dependents | 0 | crates.io database dump | Yes | |
| Active dependents as a share of the peak | 1 | crates.io database dump | Yes | |
| Binary downloads per release | Not measured | Yes |
MaintenanceLatest human commit: 3 years ago3 not measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Latest human commit | 2023-11-09 | GitHub repository | Yes | |
| Latest release | 2023-02-24 | crates.io database dump | Yes | |
| First release | 2022-03-01 | crates.io database dump | Yes | |
| Latest version | 2.0.0 | crates.io database dump | Yes | |
| Days the oldest outside issue waits for a reply | 1861 | GitHub repository | Yes | |
| Maintainer replies over 365 days | 0 | GitHub repository | Yes | |
| Repository archived | false | GitHub repository | Yes | |
| Repository gone | false | GitHub repository | Yes | |
| Release commit or manifest in the repo | true | crates.io package | Yes | |
| Published docs | true | crates.io database dump | Yes | |
| Latest release fails cargo check | Not measured | Yes | ||
| Build check date | Not measured | Yes | ||
| Build check toolchain | Not measured | Yes |
SecurityUnpatched vulnerabilities: none
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Unpatched vulnerabilities past the grace window | none | RustSec advisory database | Yes | |
| RustSec unmaintained advisory | none | RustSec advisory database | Yes | |
| Deprecation notice | none | crates.io database dump | Yes | |
| Deprecated requirement | structopt (RUSTSEC-2022-0104, https://rustsec.org/advisories/RUSTSEC-2022-0104.html) | crates.io database dump | Yes |
Distribution4 releases, 1 downloads over 30 days5 not measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Releases | 4 | crates.io database dump | Yes | |
| Downloads over 30 days | 1 | crates.io database dump | Yes | |
| Binary downloads over the last 3 releases | Not measured | Yes | ||
| Homebrew installs on request over 365 days | Not measured | Yes | ||
| Repology repository families | Not measured | Yes | ||
| Docker pull growth over 90 days | Not measured | Yes | ||
| Docker pulls at the growth baseline | Not measured | Yes | ||
| Every version a prerelease | false | crates.io database dump | Yes | |
| Every version 0.0.x | false | crates.io database dump | Yes | |
| Every version yanked | false | crates.io database dump | Yes | |
| Crate deleted from crates.io | false | crates.io database dump | Yes |
Repository178 stars, 17 forks, 5 contributors, 419 commits. Stars alone never qualify a project.
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| GitHub stars | 178 | GitHub repository | Yes — one part of the repository route | |
| GitHub forks | 17 | GitHub repository | Yes — one part of the repository route | |
| Contributors with a GitHub account | 5 | GitHub contributors | Yes — one part of the repository route | |
| Commits on the default branch | 419 | GitHub repository | No | |
| Repository's main crate left out of the entry | none | crates.io database dump | Yes |
Facts that never count
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| License | Apache-2.0 | crates.io database dump | No | |
| Runs on, from listed crates and their direct dependencies | GPU support unknown | crates.io database dump | No | |
| What shows each backend | no dependency or feature matches a backend row | crates.io database dump | No | |
| Native code, from listed crates and their dependencies | Binds native code: zstd-sys | crates.io database dump | No | |
| Python bindings, from direct dependencies of listed crates | No Python binding crate found | crates.io database dump | No | |
| Crates that depend on a listed crate directly | 0 | crates.io database dump | No |
Corrections and review history
No correction applies to this entry.
Judgments
A judgment enters only through an approved issue and a merged pull request.
No judgment applies to this entry.