llm_client

use lmcpp::*;

fn main() -> LmcppResult<()> {
    let server = LmcppServerLauncher::builder()
        .server_args(
            ServerArgs::builder()
                .hf_repo("bartowski/google_gemma-3-1b-it-qat-GGUF")?
                .build(),
        )
        .load()?;

    let res = server.completion(
        CompletionRequest::builder()
            .prompt("Tell me a joke about Rust.")
            .n_predict(64),
    )?;

    println!("Completion response: {:#?}", res.content);
    Ok(())
}

# With default model
cargo run --bin lmcpp-server-cli -- --webui
# Or with a specific model from URL:
cargo run --bin lmcpp-server-cli -- --webui -u https://huggingface.co/bartowski/google_gemma-3-1b-it-qat-GGUF/blob/main/google_gemma-3-1b-it-qat-Q4_K_M.gguf
# Or with a specific local model:
cargo run --bin lmcpp-server-cli -- --webui -l /path/to/local/model.gguf

How It Works

Your Rust App
      │
      ├─→ LmcppToolChain        (downloads / builds / caches)
      │         ↓
      ├─→ LmcppServerLauncher   (spawns & monitors)
      │         ↓
      └─→ LmcppServer           (typed handle over UDS*)
                │
                ├─→ completion()       → text generation
                └─→ other endpoints    → stuff

Endpoints ⇄ Typed Helpers

¹ Internal helper for server health.

Supported Platforms

What happened to llm_client?

And llm_devices, llm_testing, llm_prompt, llm_models, and the other crates that used to be in this repo?

• I moved cross country and took a long time off.
• Supporting local and cloud models exploded complexity.
• I realized the goals of llm_client and the goals of most people did not overlap; most people just want an Open AI compatible endpoint. They didn't want a new DSL for building AI agents or low level workflow builders.

So, I decided to narrow my scope, and start fresh. The new goal of this project is to be the best Llama.cpp integration possible.

And so, this repo will stick to the barebones and low level LLM implementation details. Shortly I will rework llm_prompt, and llm_models towards this goal.

Any further tooling built on top of that, will be a different project, which I will link to here once published.

Contact

Shelby Jenkins - Here or Linkedin