lele

Models

All of the above also run in the browser via WebAssembly.

Performance

Reference measurements, single thread, Apple M2 Pro, against ONNX Runtime (CPU, intra_op_num_threads=1, inter_op_num_threads=1). Speech models use steady-state RTF (warmup plus multi-run average); lower is better.

Kernel-level benchmarks: cargo bench --bench kernels.

Quick start

# Cargo.toml
[dependencies]
lele = "0.2"

Generate from a build script (recommended)

# model.toml
[model]
source = "hf-hub"
repo = "your-org/your-model"
files = [{ file = "model.onnx" }]

[codegen]
class_name = "MyModel"

// build.rs
fn main() -> lele_build::Result {
    // Downloads the model and writes src/mymodel.rs and
    // src/mymodel_weights.bin unless they are up to date.
    lele_build::build_model("model.toml", "src")
}

let weights = std::fs::read("src/mymodel_weights.bin")?;
let model = MyModel::new(&weights);
let output = model.forward(input);

build_models handles several models from one model.toml. HF_ENDPOINT selects a mirror; LELE_SKIP_MODEL_GEN=1 skips generation and LELE_FORCE_REGEN=1 forces it. A download or codegen error fails the build.

Or use the CLI

cargo install --git https://github.com/miuda-ai/lele --bin lele_gen
lele_gen model.onnx src/ MyModel

This writes src/mymodel.rs and src/mymodel_weights.bin. --shape input=1,512, --dim batch_size=1 and --value sr=16000 compile the model for fixed inputs (see below).

Compiler

The compiler builds a typed IR and runs its passes on it: constant and shape folding, dead-code and common-subexpression elimination, operator fusion and buffer allocation. Fusions include activations into convolutions and products, SwiGLU feed-forwards, RMSNorm, ScaleNorm pairs, rotary embeddings, SimAM, Snake, QDQ pairs around integer matmuls, and fused quantized linears.

Fixed inputs let it fold more and size the workspace up front:

[codegen.shapes]   # input shapes by name
input = [1, 512]
state = [2, 1, 128]
[codegen.dims]     # named symbolic dimensions
batch_size = 1
[codegen.values]   # inputs compiled in as constants
sr = 16000

The same settings exist on the Compiler builder (with_input_shape, with_dim, with_input_value). When every shape is known, <Model>Workspace::BYTES is the exact workspace size.

Models that share weights (e.g. a prefill and a decode step of the same transformer) can share one weights file with Compiler::with_shared_weights.

Extending it

use lele::compiler::{Compiler, Optimizations};

let compiler = Compiler::new()
    .with_name("MyModel")
    .with_default_optimizations()
    .with_pass_before_fusion(|graph| { /* rewrite the IR */ })
    .with_override("MyCustomOp", |codegen, w| {
        // Emit Rust code for this operator.
        Ok(())
    });

// Pick optimizations individually:
let compiler = Compiler::new().with_optimizations(Optimizations {
    operator_fusion: false,
    ..Optimizations::all()
});

Examples

./run_sensevoice.sh                 # SenseVoice ASR
./run_silero.sh                     # Silero VAD
./run_supertonic.sh                 # Supertonic 2 TTS
./run_supertonic3.sh                # Supertonic 3 TTS
./run_yolo26.sh                     # Yolo26 detection
./run_yolo26n_seg.sh                # Yolo26n-Seg segmentation
./run_hojo_tts.sh "text"            # Hojo-TTS-Light voice cloning

cargo run --release -p ppocr-example -- image.png
cargo run --release -p fireredvad-example -- in.wav
cargo run --release -p tone-example --bin tone -- in.wav
cargo run --release -p whisper-example --bin whisper -- in.wav --lang en

The browser demo (ASR, TTS, detection, segmentation) lives in examples/web-demo; see its README.

Supported ONNX operators

70+ operators, including:

• Math: Add, Sub, Mul, Div, Mod, Pow, Sqrt, Neg, Exp, Log, Sin, Cos, Erf, Clip, Round, CumSum, Max, Range, Einsum (partial)
• NN: Conv, ConvTranspose, ConvInteger, Gemm, MatMul, MatMulInteger, LSTM, GRU, BatchNormalization, InstanceNormalization, LayerNormalization, MaxPool, AveragePool, GlobalAveragePool, Resize
• Activation: Relu, LeakyRelu, Elu, Sigmoid, Tanh, HardSigmoid, PRelu, Softmax, LogSoftmax, ArgMax, IsNaN
• Tensor: Reshape, Transpose, Concat, Split, Slice, Gather, GatherElements, ScatterND, ScatterElements, Pad, Expand, Tile, Where, TopK, Trilu, Flatten, Squeeze, Unsqueeze
• Reduction: ReduceSum, ReduceMean, ReduceMax, ReduceL2
• Comparison / logic: Equal, Less, Greater, LessOrEqual, GreaterOrEqual, And, Or, Xor, Not
• Quantization: DynamicQuantizeLinear, QuantizeLinear, DequantizeLinear, QLinearMatMul, FakeQuantize
• Control flow / signal: If, STFT
• Other: Shape, Size, Cast, Constant, ConstantOfShape, Identity

Loop is not supported. SiLU is fused into its sigmoid and multiply.

Architecture

ONNX model ──▶ lele-build ──▶ Compiler ──▶ <model>.rs + <model>_weights.bin
                 (download,    (typed IR,        (pure Rust, no runtime deps)
                  cache)        folding,              │
                                fusion,      ┌────────┼────────┐
                                allocation)  ▼        ▼        ▼
                                          Native     WASM    Embedded
                                        (NEON/AVX) (SIMD128)

License

MIT