llmserve

From source

cargo install --path .

Usage

llmserve

The TUI has three panels:

Focus cycles between visible panels with Tab. Resize the focused panel with Shift+Left/Shift+Right.

Keybindings

Serve dialog

When you press Enter on a model, a confirmation dialog opens:

The dialog shows the resolved preset for the selected backend (context size, flash attention, batch size, GPU layers, extra args).

Features

• Auto-detects inference backends — llama-server, KoboldCpp, LocalAI, MLX (Apple Silicon), Ollama, vLLM, LM Studio
• Source tree — collapsible file tree showing all model locations with model counts and green dots for serving models
• Add directories live — press a, type a path with tab-completion, and the directory is scanned immediately and persisted to config
• Filter by source — click a source in the tree to show only its models
• Per-backend presets — context size, batch size, GPU layers, threads, and extra CLI args per backend
• Serve multiple models — run different models on different backends simultaneously, each on its own auto-assigned port
• Live log output — stdout/stderr from inference backends streams into the logs panel in real-time with color-coded error/warning highlighting
• Crash diagnostics — when a server exits, its logs are preserved so you can see exactly what went wrong
• Word wrap — press w to wrap long log lines in the logs panel
• Resizable panels — Shift+arrows to grow/shrink any focused panel
• Toggleable panels — 1 hides/shows sources, 3 hides/shows logs
• 7 themes — Default, Dracula, Solarized, Nord, Monokai, Gruvbox, Catppuccin Mocha
• Vision model support — auto-detects mmproj projector files and passes --mmproj to llama-server

Configuration

Config lives at ~/.config/llmserve/config.toml. Created automatically on first run.

# Extra directories to scan for model files
extra_model_dirs = [
    "/path/to/more/models",
]

# Global defaults
preferred_port = 8080
preferred_host = "0.0.0.0"
default_ctx_size = 8192
flash_attn = true

# Preferred backend on startup (auto-detected if not set)
# default_backend = "llama-server"

# theme = "Dracula"

Backend presets

Each backend has its own preset that overrides global defaults. Missing fields fall back to the global value.

[presets.llama-server]
ctx_size = 8192
flash_attn = true
batch_size = 2048
gpu_layers = -1          # -1 = all layers to GPU
threads = 8
extra_args = ["--mlock", "--cont-batching"]

[presets.koboldcpp]
ctx_size = 8192
gpu_layers = -1
port = 5001

[presets.localai]
ctx_size = 8192
port = 8080

[presets.mlx]
ctx_size = 4096
port = 8081

Backend detection

llmserve detects 7 backends at startup. Backends that can serve local model files are marked with a checkmark:

Backends that can't serve local files (Ollama, vLLM, LM Studio) are detected but show a clear reason in the serve dialog. They use their own model registries or manage their own servers.

Model discovery

Development

make build       # Debug build
make test        # Unit + integration tests (CI-safe)
make test-local  # All tests including local model serve rotation
make clippy      # Lint
make fmt         # Format
make install     # Install to ~/.cargo/bin

Project structure

src/
  main.rs       — Terminal init/restore, main loop
  lib.rs        — Module exports for integration tests
  app.rs        — App state, input modes, navigation, filtering, serve lifecycle
  backends.rs   — Backend detection (7 backends: llama-server, KoboldCpp, LocalAI, MLX, Ollama, vLLM, LM Studio)
  config.rs     — Config + per-backend presets, load/save TOML
  events.rs     — Crossterm event handling, vim-style keybindings
  models.rs     — Model discovery from disk + APIs
  server.rs     — Server launch/monitor/stop, non-blocking log capture
  theme.rs      — 7 color themes
  ui.rs         — Ratatui rendering (3-panel layout, popups, log viewer)
tests/
  serve_integration.rs — Integration tests (serve, verify HTTP, rotate backends)

Companion to llmfit

llmserve is designed as a companion to llmfit:

• llmfit answers: "Which models fit my hardware?" — scores models across quality, speed, fit, and context
• llmserve answers: "Let me serve this one right now" — picks a model, picks a backend, launches it

Both share the same TUI patterns (vim keys, ratatui, crossterm) and theme system.

License

MIT