rllm

Base crate is : LLM

Key Features

• Multi-backend: Manage OpenAI, Anthropic, Ollama, DeepSeek, xAI, Phind, Groq and Google through a single entry point.
• Multi-step chains: Create multi-step chains with different backends at each step.
• Templates: Use templates to create complex prompts with variables.
• Builder pattern: Configure your LLM (model, temperature, max_tokens, timeouts...) with a few simple calls.
• Chat & Completions: Two unified traits (ChatProvider and CompletionProvider) to cover most use cases.
• Extensible: Easily add new backends.
• Rust-friendly: Designed with clear traits, unified error handling, and conditional compilation via features.
• Validation: Add validation to your requests to ensure the output is what you expect.
• Evaluation: Add evaluation to your requests to score the output of LLMs.
• Parallel Evaluation: Evaluate multiple LLM providers in parallel and select the best response based on scoring functions.
• Function calling: Add function calling to your requests to use tools in your LLMs.
• REST API: Serve any LLM backend as a REST API with openai standard format.
• Vision: Add vision to your requests to use images in your LLMs.
• Reasoning: Add reasoning to your requests to use reasoning in your LLMs.
• Structured Output: Request structured output from certain LLM providers based on a provided JSON schema.
• Speech to text: Transcribe audio to text

Examples

Go to LLM Examples