higgs

Why Higgs

• Fast on Apple Silicon. Native MLX inference with prefix caching, continuous batching for transformer families, quantized KV cache, and MoE prefill tuning. See Performance.
• One endpoint for everything. Local models and remote providers behind one URL, with regex routes, aliases, and an optional auto-router that picks a model per prompt.
• Drop-in for your tools. higgs shellenv exports OPENAI_BASE_URL and ANTHROPIC_BASE_URL; Claude Code, Aider, and any OpenAI or Anthropic client just work.
• You can see what is happening. Time to first token, decode tokens per second, and prefix-cache hits for local streaming requests, plus latency, throughput, and memory, in the /metrics API and the desktop app.

Install

brew install panbanda/brews/higgs
brew install --cask panbanda/brews/higgs-desktop # desktop app; bundles the CLI

Or grab the binary and the desktop app from the latest release, or build from source with Rust 1.88+ and the Xcode Command Line Tools:

cargo build --release

Quick start

Serve a model from Hugging Face (downloaded on first use):

higgs serve --model mlx-community/Qwen3.6-35B-A3B-4bit

Talk to it with any OpenAI client:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"mlx-community/Qwen3.6-35B-A3B-4bit","messages":[{"role":"user","content":"Write one sentence about Cape Town."}]}'

Point your tools at it:

eval "$(higgs shellenv)"      # exports OPENAI_BASE_URL and ANTHROPIC_BASE_URL
higgs exec -- claude          # or aider, or anything OpenAI/Anthropic compatible

For a persistent setup with several models, providers, and routes, create a config and run the daemon:

higgs init                    # writes ~/.config/higgs/config.toml
higgs doctor                  # validates models, providers, and settings
higgs start                   # background daemon with metrics
higgs attach                  # terminal dashboard
higgs ui                      # desktop app, if installed

Desktop app

Ships with every release as a .dmg for Apple Silicon (unsigned; right-click, Open on first launch). It runs on the machine that runs Higgs and reads the same config, metrics log, and pid files as the CLI.

• Signal strip and Overview. Requests per minute, error rate, MLX memory, prefix-cache hit rate, TTFT p95 against a threshold, decode tokens per second, latency percentiles, and a live log.
• Models, Requests, Providers & Routing. Everything higgs attach shows, with TTFT, decode rate, and cache hits per model and per request, and a Hub for browsing and downloading mlx-community checkpoints.
• Chat with a request trace. Every reply keeps a trace: a waterfall of prefill (with cache hits), thinking, generation, tool calls, and stalls across rounds; throughput; tokens; the exact request with Replay and Copy as curl; raw SSE chunks; and a Copy as Markdown report for sharing speed numbers.
• Config editor. Form or raw TOML, validated on save, followed by higgs doctor, with daemon Start, Stop, and Restart.

Details, data sources, and browser-mode development: docs/desktop.md.

Supported models

MLX checkpoints from Hugging Face IDs or local paths: Qwen 3.x (dense and MoE, including 3.5, 3.6, and 3.8), Llama, Mistral, Gemma 2, 3, and 4, Phi-3, Starcoder2, DeepSeek-V2, and LLaVA-Qwen2 vision models. Unknown newer versions of a supported family load with the nearest adapter and a warning; unknown families are rejected with the supported list. The full matrix is in docs/models.md.

Configuration

[server]
host = "127.0.0.1"
port = 8000

[[models]]
path = "mlx-community/Qwen3.6-35B-A3B-4bit"
name = "qwen"

[provider.anthropic]
url = "https://api.anthropic.com"
format = "anthropic"

[[routes]]
pattern = "claude-.*"
provider = "anthropic"

[default]
provider = "higgs"

Requests for qwen run locally; anything matching claude-.* is proxied to Anthropic with format translation; everything else falls through to the default. Routing precedence, the auto-router, KV cache and prefill options, profiles, and metrics settings are covered in docs/configuration.md.

API and CLI Overview

API endpoints

• OpenAI: /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models
  • Streaming requests may set "return_progress": true to receive llama.cpp-compatible prompt_progress chunks ({total, cache, processed, time_ms}) during chunked prefill.
• Anthropic: /v1/messages, /v1/messages/count_tokens
• Metrics: /metrics (window totals, latency and time-to-first-token percentiles, aggregate tokens/s, per-model and per-provider groups with TTFT, tokens/s, and prefix-cache hits). Local streaming requests record ttft_ms and cached_tokens in the metrics JSONL log.
• System: /v1/system (version, pid, uptime, unified memory, process RSS, MLX active/peak/cache memory, loaded models with engine kind and profile)
• Health: /health

/health, /metrics, /v1/models, and /v1/system are dashboard endpoints and never count as traffic in metrics.

Core commands

• higgs serve: start in the foreground
• higgs start: start a background daemon from config or profile
• higgs stop: stop a running daemon, or use higgs stop --force
• higgs attach: open the daemon metrics dashboard
• higgs ui: open the desktop app if it is installed (macOS)
• higgs init: create ~/.config/higgs/config.toml
• higgs doctor: validate config, model paths, and providers
• higgs shellenv: print ANTHROPIC_BASE_URL and OPENAI_BASE_URL after verifying the server is reachable
• higgs exec -- <cmd>: run a tool with those variables set after the same reachability check

Apple Silicon Notes

• Release artifacts bundle mlx.metallib.
• Source builds also require mlx.metallib next to the executable. Higgs now restores it automatically from Cargo build output when possible, then fails loudly if it still cannot be found.
• [local].raise_wired_limit defaults to false. Enable it only when you explicitly want MLX to raise the process wired-memory limit.
• batch=true is only supported for transformer families with true batched decode support.
• For batch models, prefill_yield_tokens can interleave long prompt prefills with decode; use 0 or omit it for the synchronous default.

Performance

Benchmarks below were run on M4 Max 128GB. Methodology, harness details, and benchmark-driven defaults are documented in docs/benchmarking.md.

Decode Throughput (tok/s)

Single request, 500 generated tokens, median of 3 runs.

MLX models use 4-bit, or 8-bit for MoE. llama.cpp and Ollama use Q4_K_M, or Q8_0 for MoE.

MoE Prefill (time to first token)

Measured on DeepSeek-V2-Lite-4bit with global batch sorting before gather_qmm.

Continuous Batching (Llama-1B)

Memory (RSS in MB)

Feature Comparison

Development

cargo test -- --test-threads=1
cargo clippy
cargo fmt --check
cd apps/desktop && pnpm install && pnpm typecheck && pnpm build

Contributor workflow, project structure, release validation, and doc expectations live in CONTRIBUTING.md. Migration notes for older higgs start flags are in docs/configuration.md.

License

MIT