llm

Key Features

• Multi-backend: Manage OpenAI, Anthropic, Ollama, DeepSeek, xAI, Phind, Groq, OpenRouter, Cohere, Elevenlabs and Google through a single entry point.
• Multi-step chains: Create multi-step chains with different backends at each step.
• Templates: Use templates to create complex prompts with variables.
• Builder pattern: Configure your LLM (model, temperature, max_tokens, timeouts...) with a few simple calls.
• Chat & Completions: Two unified traits (ChatProvider and CompletionProvider) to cover most use cases.
• Extensible: Easily add new backends.
• Rust-friendly: Designed with clear traits, unified error handling, and conditional compilation via features.
• Validation: Add validation to your requests to ensure the output is what you expect.
• Resilience (retry/backoff): Enable resilient calls with exponential backoff and jitter.
• Evaluation: Add evaluation to your requests to score the output of LLMs.
• Parallel Evaluation: Evaluate multiple LLM providers in parallel and select the best response based on scoring functions.
• Function calling: Add function calling to your requests to use tools in your LLMs.
• REST API: Serve any LLM backend as a REST API with openai standard format.
• Vision: Add vision to your requests to use images in your LLMs.
• Reasoning: Add reasoning to your requests to use reasoning in your LLMs.
• Structured Output: Request structured output from certain LLM providers based on a provided JSON schema.
• Speech to text: Transcribe audio to text
• Text to speech: Transcribe text to audio
• Memory: Store and retrieve conversation history with sliding window (soon others) and shared memory support
• Agentic: Build reactive agents that can cooperate via shared memory, with configurable triggers, roles and validation.

Use any LLM backend on your project

Simply add LLM to your Cargo.toml:

[dependencies]
llm = { version = "1.3.8", features = ["openai", "anthropic", "ollama", "deepseek", "xai", "phind", "google", "groq", "mistral", "elevenlabs"] }

Use any LLM on cli

LLM includes a command-line tool for easily interacting with different LLM models. You can install it with: cargo install llm

• Use llm to start an interactive chat session
• Use llm openai:gpt-4o to start an interactive chat session with provider:model
• Use llm set OPENAI_API_KEY your_key to configure your API key
• Use llm default openai:gpt-4 to set a default provider
• Use echo "Hello World" | llm to pipe
• Use llm --provider openai --model gpt-4 --temperature 0.7 for advanced options

Serving any LLM backend as a REST API

• Use standard messages format
• Use step chains to chain multiple LLM backends together
• Expose the chain through a REST API with openai standard format

[dependencies]
llm = { version = "1.3.8", features = ["openai", "anthropic", "ollama", "deepseek", "xai", "phind", "google", "groq", "api", "mistral", "elevenlabs"] }

More details in the api_example

More examples

Usage

Here's a basic example using OpenAI for chat completion. See the examples directory for other backends (Anthropic, Ollama, DeepSeek, xAI, Google, Phind, Elevenlabs), embedding capabilities, and more advanced use cases.

use llm::{
    builder::{LLMBackend, LLMBuilder}, // Builder pattern components
    chat::ChatMessage,                 // Chat-related structures
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Get OpenAI API key from environment variable or use test key as fallback
    let api_key = std::env::var("OPENAI_API_KEY").unwrap_or("sk-TESTKEY".into());

    // Initialize and configure the LLM client
    let llm = LLMBuilder::new()
        .backend(LLMBackend::OpenAI)	// Use OpenAI as the LLM provider
        .api_key(api_key) 						// Set the API key
        .model("gpt-4.1-nano") 				// Use GPT-4.1 Nano model
        .max_tokens(512) 							// Limit response length
        .temperature(0.7) 						// Control response randomness (0.0-1.0)
        .normalize_response(true)     // Increase response normalization (e.g. function call stream)
        .build()
        .expect("Failed to build LLM");

    // Prepare conversation history with example messages
    let messages = vec![
        ChatMessage::user()
            .content("Tell me that you love cats")
            .build(),
        ChatMessage::assistant()
            .content("I am an assistant, I cannot love cats but I can love dogs")
            .build(),
        ChatMessage::user()
            .content("Tell me that you love dogs in 2000 chars")
            .build(),
    ];

    // Send chat request and handle the response
    match llm.chat(&messages).await {
        Ok(response) => {
            // Print the response text
            if let Some(text) = response.text() {
                println!("Response: {text}");
            }
            // Print usage information
            if let Some(usage) = response.usage() {
                println!("  Prompt tokens: {}", usage.prompt_tokens);
                println!("  Completion tokens: {}", usage.completion_tokens);
            } else {
                println!("No usage information available");
            }
        }
        Err(e) => eprintln!("Chat error: {e}"),
    }
    Ok(())
}