rust-dl-webserver
App Listed Dormant
🦀 Example of serving deep learning models in Rust with batched prediction
- Downloads
- Not measured
- Dependents
- Not applicable
- Stars
- 34
- Forks
- 5
- Contributors
- 3
- Commits
- 70
About
This project provides an example of serving a deep learning model with batched prediction using Rust. In particular it runs a GPT2 model from rust-bert to generate text based on input context.
Features
- Batched prediction using
batched-fnwhen GPU is detected. - Back pressure mechanism that will return a 503 status code if the server gets back-logged with too many requests.
Running the server
In order for the server to make use of your GPU (if you have one available) you'll need to compile it against the right version of the LibTorch C++ library, which you can download from https://pytorch.org/get-started/locally/.
After downloading, unzip the file and then run the server with
make run LIBTORCH=/path/to/libtorch
If you don't have a GPU available, just run
make run
Now in a separate terminal you can send several requests in to the server at once:
Read the full README
curl -d '{"text":"Hello, World!"}' \
-H "Content-Type: application/json" \
http://localhost:3030/generate &
curl -d '{"text":"Stay at home"}' \
-H "Content-Type: application/json" \
http://localhost:3030/generate &
curl -d '{"text":"Wash your hands"}' \
-H "Content-Type: application/json" \
http://localhost:3030/generate &
curl -d '{"text":"Do not touch your face"}' \
-H "Content-Type: application/json" \
http://localhost:3030/generate &
The logs from the server should look something like this:
Compatibility
- License
- None stated
- Crate facts
- Not measured
- Runs on, Native code and Python bindings read crates on crates.io. This entry lists no crate.
Compare with similar projects
Order: strongest tier first, then most downloads, then name, with unmeasured values last. A library counts downloads over 30 days. An app counts binary downloads per release. An app with none counts its crate downloads over 30 days. ML Rust runs this catalogue, so its closest project always gets a row.
| Project | Downloads | Dependents | Stars | Latest release | Runs on (GPU) | Native code | License |
|---|---|---|---|---|---|---|---|
|
🦀 Example of serving deep learning models in Rust with batched prediction |
Not measured | Not applicable | 34 | Not measured | Not measured | Not measured | None stated |
|
ort Featured
A safe Rust wrapper for ONNX Runtime 1.31 - Optimize and accelerate machine learning inference & training |
3,902,768 downloads over 30 days | 694 | 2,611 | 2.0.0-rc.15 | CUDA, WebGPU, ROCm | Binds native code: onnxruntime, ort-sys. Build uses a native toolchain: pkg-config (feature pkg-config) | Apache-2.0 OR MIT |
|
llama-gguf Listed
A high-performance Rust implementation of llama.cpp - LLM inference engine with full GGUF support |
1,787 downloads over 30 days | 3 | 21 | 0.14.0 | CUDA, Metal, Vulkan | Binds native code: hailort-sys (feature hailo), libsqlite3-sys (feature rag-sqlite) | Apache-2.0 OR MIT |
|
TUI for serving local LLM models. Pick a model, pick a backend, serve it. |
424 binary downloads per release | Not applicable | 417 | 0.0.7 | GPU support unknown | No native code found | MIT |
|
lele Listed
Bare-metal machine learning inference library in Rust |
132 downloads over 30 days | 1 | 183 | 0.2.0 | GPU support unknown | No native code found | MIT |
|
Middleware for serving deep learning models with batched prediction |
76 downloads over 30 days | 0 | 22 | 0.2.5 | GPU support unknown | No native code found | Apache-2.0 |
|
Inference server for LLMs, speech and embeddings: OpenAI-compatible routes with streaming, plus speech synthesis (TTS) and transcription (ASR). Serves Llama, Mistral, Qwen, Mamba2, MLA+MoE and hybrid models from SafeTensors, AWQ, GPTQ and GGUF. |
8 binary downloads per release | Not applicable | 7 | 0.7.1 | CUDA, WebGPU | Binds native code: ring (feature distributed) | Apache-2.0 |
Why ListedNo archive, New or Featured rule places it, so it is Listed.
What it lacks for Featured
- The project is dormant and not finished. Dormancy rule
What else the check found
- The last activity is on 8 Mar 2021, 2072 days ago. That reaches the 365 days the rules expect, so the project is dormant. Dormancy rule
Not measured: Latest release, Unpatched vulnerabilities past the grace window, Binary downloads per release, First release.
Exact rules: Listed, Featured app. The check on applied thresholds version 9.
Every rule outcome, as the check records it
expected-interval: app: E = 365 daysdormancy: last activity 2021-03-08, 2072 days ago, E = 365: dormantfeatured: not featured: dormant and not stablelisted: listed: no archive, new or featured rule applies
Measurements and sources
Each signal row shows the raw value, its source and the fetch date. The Sources list gathers every distinct source on this page. Movement needs more history.
Sources4 sources
- Description: the GitHub repository description
- GitHub repository, fetched
- GitHub releases, fetched
- GitHub contributors, fetched
AdoptionNot measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Active dependents | Not measured | Yes | ||
| New dependents (365 days) | Not measured | Yes | ||
| Peak active dependents | Not measured | Yes | ||
| Active dependents as a share of the peak | Not measured | Yes | ||
| Binary downloads per release | Not measured | Yes |
MaintenanceLatest human commit: 5 years ago8 not measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Latest human commit | 2021-03-08 | GitHub repository | Yes | |
| Latest release | Not measured | Yes | ||
| First release | Not measured | Yes | ||
| Latest version | Not measured | Yes | ||
| Days the oldest outside issue waits for a reply | 0 | GitHub repository | Yes | |
| Maintainer replies over 365 days | 0 | GitHub repository | Yes | |
| Repository archived | false | GitHub repository | Yes | |
| Repository gone | false | GitHub repository | Yes | |
| Release commit or manifest in the repo | Not measured | Yes | ||
| Published docs | Not measured | Yes | ||
| Latest release fails cargo check | Not measured | Yes | ||
| Build check date | Not measured | Yes | ||
| Build check toolchain | Not measured | Yes |
SecurityNot measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Unpatched vulnerabilities past the grace window | Not measured | Yes | ||
| RustSec unmaintained advisory | Not measured | Yes | ||
| Deprecation notice | Not measured | Yes | ||
| Deprecated requirement | Not measured | Yes |
Distribution0 binary downloads over the last 3 releases10 not measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Releases | Not measured | Yes | ||
| Downloads over 30 days | Not measured | Yes | ||
| Binary downloads over the last 3 releases | 0 | GitHub releases | Yes | |
| Homebrew installs on request over 365 days | Not measured | Yes | ||
| Repology repository families | Not measured | Yes | ||
| Docker pull growth over 90 days | Not measured | Yes | ||
| Docker pulls at the growth baseline | Not measured | Yes | ||
| Every version a prerelease | Not measured | Yes | ||
| Every version 0.0.x | Not measured | Yes | ||
| Every version yanked | Not measured | Yes | ||
| Crate deleted from crates.io | Not measured | Yes |
Repository34 stars, 5 forks, 3 contributors, 70 commits. Stars alone never qualify a project.1 not measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| GitHub stars | 34 | GitHub repository | Yes — one part of the repository route | |
| GitHub forks | 5 | GitHub repository | Yes — one part of the repository route | |
| Contributors with a GitHub account | 3 | GitHub contributors | Yes — one part of the repository route | |
| Commits on the default branch | 70 | GitHub repository | No | |
| Repository's main crate left out of the entry | Not measured | Yes |
Facts that never count
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| License | None stated | GitHub repository | No | |
| Runs on, from listed crates and their direct dependencies | Not measured | No | ||
| What shows each backend | Not measured | No | ||
| Native code, from listed crates and their dependencies | Not measured | No | ||
| Python bindings, from direct dependencies of listed crates | Not measured | No | ||
| Crates that depend on a listed crate directly | Not measured | No |
Corrections and review history
No correction applies to this entry.
Judgments
A judgment enters only through an approved issue and a merged pull request.
No judgment applies to this entry.