cake

Quick Start

Build

cargo build --release --features cuda        # Linux (NVIDIA)
cargo build --release --features metal       # macOS (Apple Silicon GPU)
cargo build --release --features accelerate  # macOS (Apple Silicon CPU, F32 models)
cargo build --release --features vulkan      # Linux (AMD/Intel/Steam Deck)
cargo build --release                        # CPU only (portable)

Models

Download models from HuggingFace with cake pull. Models are stored in the standard HuggingFace cache directory (~/.cache/huggingface/hub/) and are shared with any other tools that use the same cache (transformers, huggingface-cli, etc.).

cake pull evilsocket/Qwen3-0.6B             # text model (600M params)
cake pull evilsocket/flux1-dev               # image model (FLUX.1-dev FP8)
cake pull evilsocket/VibeVoice-1.5B          # voice synthesis model

cake list                                    # show all locally available models
cake rm evilsocket/Qwen3-0.6B               # delete a cached model

Models are also downloaded automatically on first use if not already cached.

Single Node

Run any model locally on a single machine — architecture is auto-detected from the model's config.json:

# Text generation
cake run evilsocket/Qwen3-0.6B "Explain quantum computing in simple terms"

# Interactive TUI chat
cake chat Qwen/Qwen3-0.6B

# Start an API server + web UI
cake serve evilsocket/Qwen3-0.6B

# Image generation (FLUX.1-dev FP8)
cake run evilsocket/flux1-dev --model-type image-model --image-model-arch flux1 \
  "a cyberpunk cityscape at night"

# Voice synthesis with voice cloning
cake run evilsocket/VibeVoice-1.5B --model-type audio-model \
  --voice-prompt voice.wav "Hello world"

Distributed

Shard a model across multiple machines using --cluster-key. Workers don't need the model data — the master automatically streams the required tensor weights over the network (compressed with zstd, verified with CRC32 checksums). Workers cache received data locally for subsequent runs.

# Start workers on any machines (no model needed)
cake run --cluster-key mysecret --name gpu-server-1    # machine A
cake run --cluster-key mysecret --name macbook          # machine B

# Run inference from the master (has the model)
cake run evilsocket/Qwen3-0.6B "Hello" --cluster-key mysecret

# Or start an API server as the master
cake serve evilsocket/Qwen3-0.6B --cluster-key mysecret

The master discovers workers via mDNS, assigns layers proportionally to each device's VRAM/compute, and pushes only the required weight shards. See the clustering documentation for manual topology files and advanced configuration.

For the full usage guide and API reference, check the project documentation.

Star History

License

Released under the FAIR License (Free for Attribution and Individual Rights) v1.0.0.

• Non-commercial use (personal, educational, research, non-profit) is freely permitted under the terms of the license.
• Commercial use (SaaS, paid apps, any monetization) requires visible attribution to the project and its author. See the license for details.
• Business use (any use by or on behalf of a business entity) requires a signed commercial agreement with the author. Contact evilsocket@gmail.com for inquiries.

To see the licenses of the project dependencies, install cargo license with cargo install cargo-license and then run cargo license.