ollama-rs

Using with llmman

llmman is a local model runner that serves the Ollama API on port 17434, so ollama-rs works with it unchanged; just point the client at that port:

use ollama_rs::Ollama;

// llmman listens on 127.0.0.1:17434 by default
let ollama = Ollama::builder()
    .host("http://localhost")
    .port(17434)
    .build();

Start the server with llmman serve and pull a model with llmman pull gemma4 (or llmman pull hf.co/unsloth/Qwen3.5-0.8B-GGUF to pull from Hugging Face).

Usage

Feel free to check the Chatbot example that shows how to use the library to create a simple chatbot in less than 50 lines of code. You can also check some other examples.

These examples use poor error handling for simplicity, but you should handle errors properly in your code.

Completion Generation

use ollama_rs::generation::completion::request::GenerationRequest;

let model = "llama2:latest".to_string();
let prompt = "Why is the sky blue?".to_string();

let res = ollama.generate(GenerationRequest::new(model, prompt)).await;

if let Ok(res) = res {
    println!("{}", res.response);
}

OUTPUTS: The sky appears blue because of a phenomenon called Rayleigh scattering...

Completion Generation (Streaming)

Requires the stream feature.

use ollama_rs::generation::completion::request::GenerationRequest;
use tokio::io::{self, AsyncWriteExt};
use tokio_stream::StreamExt;

let model = "llama2:latest".to_string();
let prompt = "Why is the sky blue?".to_string();

let mut stream = ollama.generate_stream(GenerationRequest::new(model, prompt)).await.unwrap();

let mut stdout = io::stdout();
while let Some(res) = stream.next().await {
    let responses = res.unwrap();
    for resp in responses {
        stdout.write_all(resp.response.as_bytes()).await.unwrap();
        stdout.flush().await.unwrap();
    }
}

Same output as above but streamed.

Completion Generation (With Options)

use ollama_rs::generation::completion::request::GenerationRequest;
use ollama_rs::models::ModelOptions;

let model = "llama2:latest".to_string();
let prompt = "Why is the sky blue?".to_string();

let options = ModelOptions::default()
    .temperature(0.2)
    .repeat_penalty(1.5)
    .top_k(25)
    .top_p(0.25);

let res = ollama.generate(GenerationRequest::new(model, prompt).options(options)).await;

if let Ok(res) = res {
    println!("{}", res.response);
}

OUTPUTS: 1. Sun emits white sunlight: The sun consists primarily ...

Chat Mode

Every message sent and received will be stored in the library's history.

Example with history:

use ollama_rs::generation::chat::{request::ChatMessageRequest, ChatMessage};
use ollama_rs::history::ChatHistory;

let model = "llama2:latest".to_string();
let prompt = "Why is the sky blue?".to_string();
// `Vec<ChatMessage>` implements `ChatHistory`,
// but you could also implement it yourself on a custom type
let mut history = vec![];

let res = ollama
    .send_chat_messages_with_history(
        &mut history, // <- messages will be saved here
        ChatMessageRequest::new(
            model,
            vec![ChatMessage::user(prompt)], // <- You should provide only one message
        ),
    )
    .await;

if let Ok(res) = res {
    println!("{}", res.message.content);
}

Check chat with history examples for default and stream

List Local Models

let res = ollama.list_local_models().await.unwrap();

Returns a vector of LocalModel structs.

Show Model Information

let res = ollama.show_model_info("llama2:latest".to_string()).await.unwrap();

Returns a ModelInfo struct.

Create a Model

use ollama_rs::models::create::CreateModelRequest;

let res = ollama.create_model(CreateModelRequest::new("my_model".into())
    .system("You're a chat bot.".into())
    .from_model("llama2:latest".into())).await.unwrap();

Returns a CreateModelStatus struct representing the final status of the model creation.

Create a Model (Streaming)

Requires the stream feature.

use ollama_rs::models::create::CreateModelRequest;
use tokio_stream::StreamExt;

let res = ollama.create_model_stream(CreateModelRequest::new("my_model".into())
    .system("You're a chat bot.".into())
    .from_model("llama2:latest".into())).await.unwrap();

while let Some(res) = res.next().await {
    let res = res.unwrap();
    // Handle the status
}

Returns a CreateModelStatusStream that will stream every status update of the model creation.

Copy a Model

let _ = ollama.copy_model("mario".into(), "mario_copy".into()).await.unwrap();

Delete a Model

let _ = ollama.delete_model("mario_copy".into()).await.unwrap();

Generate Embeddings

use ollama_rs::generation::embeddings::request::GenerateEmbeddingsRequest;

let request = GenerateEmbeddingsRequest::new("llama2:latest".to_string(), "Why is the sky blue?".into());
let res = ollama.generate_embeddings(request).await.unwrap();

Generate Embeddings (Batch)

use ollama_rs::generation::embeddings::request::GenerateEmbeddingsRequest;

let request = GenerateEmbeddingsRequest::new("llama2:latest".to_string(), vec!["Why is the sky blue?", "Why is the sky red?"].into());
let res = ollama.generate_embeddings(request).await.unwrap();

Returns a GenerateEmbeddingsResponse struct containing the embeddings (a vector of floats).

System One

System One supports choice, yes/no (noul), and score questions with local decision models such as nimble. It requires Ollama 0.35.0 or later and a downloaded compatible model.

use ollama_rs::generation::systemone::{
    request::{Question, SystemOneRequest},
    SystemOneAnswer,
};

let request = SystemOneRequest::builder(
    "nimble",
    "Our checkout has returned 500 errors since 9am.",
)
.question(
    "refund",
    Question::noul("Is the customer requesting a refund?"),
)
.question(
    "urgency",
    Question::score(
        "How urgently does this ticket need a response?",
        [
            "Routine: no time pressure",
            "Soon: a customer is inconvenienced",
            "Immediate: a critical service is unavailable",
        ],
    ),
)
.question(
    "label",
    Question::choice(
        "Which label fits this ticket?",
        [
            ("billing", "Payments and refunds"),
            ("bug", "Software errors"),
            ("account", "Login and account access"),
        ],
    ),
);

let response = ollama.system_one(request).await.unwrap();

for answer in response.answers.values() {
    match answer {
        SystemOneAnswer::Choice { choice, .. } => println!("Choice: {choice}"),
        SystemOneAnswer::Noul { noul } => println!("True probability: {noul}"),
        SystemOneAnswer::Score { score, .. } => println!("Score: {score}"),
    }
}

The response contains typed choice, noul, and score answers, along with token usage. See the System One documentation for the available question types and model requirements.

Make a Function Call

use ollama_rs::coordinator::Coordinator;
use ollama_rs::generation::chat::ChatMessage;
use ollama_rs::generation::tools::implementations::{DDGSearcher, Scraper, Calculator};
use ollama_rs::models::ModelOptions;

let mut history = vec![];

let mut coordinator = Coordinator::new(ollama, "qwen2.5:32b".to_string(), history)
    .options(ModelOptions::default().num_ctx(16384))
    .add_tool(DDGSearcher::new())
    .add_tool(Scraper {})
    .add_tool(Calculator {});

let resp = coordinator
    .chat(vec![ChatMessage::user("What is the current oil price?")])
    .await.unwrap();

println!("{}", resp.message.content);

Uses the given tools (such as searching the web) to find an answer, feeds that answer back into the LLM, and returns a ChatMessageResponse with the answer to the question.

Create a custom tool

The function macro simplifies the creation of custom tools. Below is an example of a tool that retrieves the current weather for a specified city:

/// Retrieve the weather for a specified city.
///
/// * city - The city for which to get the weather.
#[ollama_rs::function]
async fn get_weather(city: String) -> Result<String, Box<dyn std::error::Error + Sync + Send>> {
    let url = format!("https://wttr.in/{city}?format=%C+%t");
    let response = reqwest::get(&url).await?.text().await?;
    Ok(response)
}

To create a custom tool, define a function that returns a Result<String, Box<dyn std::error::Error + Sync + Send>> and annotate it with the function macro. This function will be automatically converted into a tool that can be used with the Coordinator, just like any other tool.

Ensure that the doc comment above the function clearly describes the tool's purpose and its parameters. This information will be provided to the LLM to help it understand how to use the tool.

When using streaming chat directly, you may also attach tool schemas to the request:

use ollama_rs::generation::chat::request::ChatMessageRequest;

let request = ChatMessageRequest::new("lfm2.5:8b".to_owned(), Vec::new()).add_tool(get_weather);
let mut stream = ollama.send_chat_messages_stream(request).await.unwrap();

send_chat_messages_stream yields streamed ChatMessageResponse chunks. If a chunk contains message.tool_calls, run the requested tools and include their results in a follow-up request.

For a more detailed example, see the function call example.

Completion Generation (With Thinking)

let model = "qwen3:latest".to_string();
let prompt = "Why is the sky blue?".to_string();

let res = ollama.generate(GenerationRequest::new(model, prompt).think(true)).await;

if let Ok(res) = res {
    println!("{}", res.response);
}