Inference runtimes

High-level bindings for OpenVINO.

Lightweight framework for building ONNX runtime pipelines with ort

Tiny, no-nonsense, self contained, TensorFlow and ONNX inference

Wasmtime implementation of the wasi-nn API

Rust bindings for OpenNMT/CTranslate2

Generate music based on natural language prompts using LLMs running locally

A Datacenter Scale Distributed Inference Serving Framework

Token samplers for large language models

Structured Generation

Python bindings

A high-performance Rust implementation of llama.cpp - LLM inference engine with full GGUF support

Distributed AI/LLM for the people. Share compute privately or publicly to power your agents and chat.

The easiest & fastest way to run customized and fine-tuned LLMs locally or on the edge

Wrapper around Microsoft's ONNX Runtime

Pure Rust ONNX inference engine — zero C/C++ dependencies

Rust bindings for TensorFlow Lite

TUI for serving local LLM models. Pick a model, pick a backend, serve it.

Lightweight Ollama-compatible inference server with native SafeTensors support. No Python dependencies, cross-platform WebGPU acceleration via Airframe.

High level API for stable-diffusion.cpp

The all-in-one RWKV runtime box with embed, RAG, AI agents, and more.

aha model inference library, now supports Qwen(2.5VL/3/3VL/3.5/ASR/3Embedding/3Reranker), MiniCPM(4/5), VoxCPM(0.5B/1.5/2), DeepSeek-OCR/2, Hunyuan-OCR, PaddleOCR-VL/1.5, RMBG2.0, GLM(ASR-Nano-2512/OCR), Fun-ASR-Nano-2512, LFM(2/2.5/2VL/2.5VL)

AICI: Prompts as (Wasm) Programs

Efficent platform for inference and serving local LLMs including an OpenAI compatible API server.

Bare-metal machine learning inference library in Rust

Middleware for serving deep learning models with batched prediction

NobodyWho is an inference engine that lets you run LLMs locally and efficiently on any device.

A rust mlx server, ui, and model router for fast, dependency free inference on apple architecture

Distributed LLM inference for mobile, desktop and server.

Rust frontend support for TVM

Pie: A Programmable LLM Serving System

High-performance LLM fine-tuning framework for Apple Silicon

Python bindings

Inference server for LLMs, speech and embeddings: OpenAI-compatible routes with streaming, plus speech synthesis (TTS) and transcription (ASR). Serves Llama, Mistral, Qwen, Mamba2, MLA+MoE and hybrid models from SafeTensors, AWQ, GPTQ and GGUF.

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

Pure Rust 1-bit LLM inference engine for PrismML Bonsai models — umbrella crate

Pure Rust implementation of LLaMA-family of models, executable

Machine Learning Inference Framework and Server Runtime

Python bindings

Run ONNX and TensorFlow inference in the browser.

🦀 Example of serving deep learning models in Rust with batched prediction