Run and serve models

High-level bindings for OpenVINO.

High-performance document intelligence library for Rust. Extract text, metadata, and structured data from PDFs, Office documents, images, and 91+ formats and 248 programming languages via tree-sitter code intelligence with async/sync APIs.

Fast ASR & Speaker Diarization with NVIDIA Parakeet via ONNX

Lightweight framework for building ONNX runtime pipelines with ort

Inference engine for GLiNER models

An Optical Character Recognition (OCR) and Document Layout Analysis library written in Rust.

Ready-to-use NLP pipelines and language models

Tiny, no-nonsense, self contained, TensorFlow and ONNX inference

Wasmtime implementation of the wasi-nn API

Rust bindings for OpenNMT/CTranslate2

Agent Framework for Building Autonomous Agents

Generate music based on natural language prompts using LLMs running locally

A Datacenter Scale Distributed Inference Serving Framework

DORA (Dataflow-Oriented Robotic Architecture) is middleware designed to streamline and simplify the creation of AI-based robotic applications. It offers low latency, composable, and distributed dataflow capabilities. Applications are modeled as directed graphs, also referred to as pipelines.

Token samplers for large language models

Minimal ONNX model loader for extracting weights, tensors, operations, and graph structure

Ultralytics YOLO inference library and CLI for Rust

A library for running chains of LLMs (such as ChatGPT) in series to complete complex tasks, such as text summation.

GGUF file parser

Structured Generation

Python bindings

Production-oriented Rust wrapper for the Silero VAD ONNX model.

A high-performance Rust implementation of llama.cpp - LLM inference engine with full GGUF support

Fast Rust speaker diarization with pyannote-level accuracy and native CPU/CoreML/CUDA inference

Distributed AI/LLM for the people. Share compute privately or publicly to power your agents and chat.

ML framework on numr for training and inference on CPU, CUDA and WebGPU: attention, quantization, inference scheduling and distributed execution. Models include LLaMA, Mamba2, Qwen3.5, Whisper, Kokoro, VITS, VoxCPM2, Silero VAD, CLAP and super-resolution. Loads SafeTensors and GGUF.

Postgres with GPUs for ML/AI apps.

Python bindings

The easiest & fastest way to run customized and fine-tuned LLMs locally or on the edge

Wrapper around Microsoft's ONNX Runtime

A simple interface for pretrained AI models

Pure Rust ONNX inference engine — zero C/C++ dependencies

Rust bindings for TensorFlow Lite

Zero-dependency ONNX model parser in Rust

An implementation of the RWKV language model in pure WebGPU.

TUI for serving local LLM models. Pick a model, pick a backend, serve it.

A Rust library integrated with ONNXRuntime, providing a collection of ML models.

Lightweight Ollama-compatible inference server with native SafeTensors support. No Python dependencies, cross-platform WebGPU acceleration via Airframe.

High level API for stable-diffusion.cpp

A Rust implementation of the Qwen3 Text-to-Speech (TTS) model inference.

aha model inference library, now supports Qwen(2.5VL/3/3VL/3.5/ASR/3Embedding/3Reranker), MiniCPM(4/5), VoxCPM(0.5B/1.5/2), DeepSeek-OCR/2, Hunyuan-OCR, PaddleOCR-VL/1.5, RMBG2.0, GLM(ASR-Nano-2512/OCR), Fun-ASR-Nano-2512, LFM(2/2.5/2VL/2.5VL)

AICI: Prompts as (Wasm) Programs

Efficent platform for inference and serving local LLMs including an OpenAI compatible API server.

Bare-metal machine learning inference library in Rust

High-performance model-routing gateway for large-scale LLM deployments

Middleware for serving deep learning models with batched prediction

Rust implementation of the Diffusers library using Torch.

llm_client: The easiest Rust interface for local LLMs

NobodyWho is an inference engine that lets you run LLMs locally and efficiently on any device.

One backbone, many specialists. CMF is a self-describing, memory-mappable single-file format and a dependency-free runtime for quantized LLMs — zero-copy execution on CPU or GPU (Vulkan · Metal · DX12) that overlays task-specialized skills onto one shared model, with no per-model duplication.

A rust mlx server, ui, and model router for fast, dependency free inference on apple architecture

Distributed LLM inference for mobile, desktop and server.

Rust frontend support for TVM

Pie: A Programmable LLM Serving System

High-performance LLM fine-tuning framework for Apple Silicon

Python bindings

A local AI chat app powered by 🦀Rust, 🧊iced, 🤗Hugging Face, and 🦙llama.cpp

Rust implementation of Sentence Bert (SBert)

Inference server for LLMs, speech and embeddings: OpenAI-compatible routes with streaming, plus speech synthesis (TTS) and transcription (ASR). Serves Llama, Mistral, Qwen, Mamba2, MLA+MoE and hybrid models from SafeTensors, AWQ, GPTQ and GGUF.

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Rust port of KittenTTS — ultra-lightweight ONNX-based text-to-speech

🦀 A Pure Rust Framework For Building AGIs.

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

Pure Rust 1-bit LLM inference engine for PrismML Bonsai models — umbrella crate

Pure Rust implementation of LLaMA-family of models, executable

Machine Learning Inference Framework and Server Runtime

Python bindings

Run ONNX and TensorFlow inference in the browser.

🦀 Example of serving deep learning models in Rust with batched prediction