mel_spec

Main Features

Quick Start

use mel_spec::prelude::*;

let samples = vec![0.0_f32; 16_000];
let mel_frames = Spectrogram::compute_mel_spectrogram_cpu(
    &samples,
    400,
    160,
    80,
    16_000.0,
);

println!("frames={}", mel_frames.len());

For whole utterances, BatchLogMelSpectrogram gives feature-major output. Keep the scratch buffers to reuse them across calls:

use mel_spec::prelude::*;

let frontend = BatchLogMelSpectrogram::new(BatchLogMelConfig::default()).unwrap();
let mut scratch = frontend.scratch();
let samples = vec![0.0_f32; 16_000];
let features = frontend.compute_with_scratch(&samples, &mut scratch).unwrap();

assert_eq!(features.shape(), &[80, 101]);

Use BatchLogMelSpectrogram::compute_interleaved for interleaved audio. The method averages all input channels before feature extraction.

The focused API examples are kept in the example READMEs so the top-level README stays readable:

Voice Activity Detection

mel-spec includes a lightweight, model-free VAD. It does not load a neural VAD runtime. It looks for speech-like Sobel edge structure in mel spectrogram frames and can attach STFT-derived timestamps to each decision.

Current balanced default on the checked-in TEN-VAD testset:

The balanced default is not trying to beat learned VADs at strict endpointing. It is a fast built-in option that reuses ASR mel features and avoids another model dependency. Tuned Silero is still more accurate overall. TEN-VAD is the source of the labels and upstream reports stronger precision/recall than Silero and WebRTC on the same testset.

Detailed method, provenance, commands, speed notes, and per-file results are in doc/vad/README.md.

TGA Spectrograms

TGA spectrograms are useful when you want a simple interchange format for mel features. They can be inspected as images, spliced, stored, and passed to the Whisper examples without keeping the original audio around.

This path is live in Hush as local browser ASR. The browser uses the Whisper-compatible log-mel output from mel-spec. It stores captured speech as compact 8-bit TGA images. It decodes the images to an 80-mel Float32Array and passes the tensor to whisper_set_mel in a custom whisper.cpp WASM binding. The active Hush deployment verifies that local WASM Whisper can transcribe from the mel tensor without posting microphone audio to a server. This uses the proposed direct-mel entry point for whisper.cpp, not the stock browser example that feeds PCM audio into whisper.wasm.

"the quest for peace."

Mel spectrograms are also robust under heavy quantization. Whisper does not need high-precision PCM after projection into mel space. Eight-bit TGA images preserve the information that the model sees. Coarse mel-value rounding can also retain useful transcription quality.

Original: [0.158, 0.266, 0.076, 0.196, 0.167, ...]
Rounded:  [0.2,   0.3,   0.1,   0.2,   0.2,   ...]

(top: original mel values, bottom: values rounded to 1.0e-1 before image quantization)

Performance

mel-spec 0.5.0 on the 11-second JFK sample, one Apple M1 core, release build:

The Parakeet configuration has 128 mels, pre-emphasis 0.97, and per-feature normalization. The time increases linearly with the audio length: 66 seconds of audio take 6.8 ms in the Parakeet configuration.

mel() and the Kaldi filterbank builder produce dense filterbank matrices for reference, fixture comparison, and interchange with other toolchains. The CPU paths execute a sparse form of the same matrices, with one contiguous band of weights for each mel row. Tests compare the sparse projection with the dense matrices.

Parakeet/NeMo Frontend Comparison

asr-api uses BatchLogMelSpectrogram for its Parakeet/TDT frontend. The parakeet_featurizer_bench harness in asr-torch compares this frontend with a CPU TorchScript trace of the NeMo Parakeet featurizer (featurizer_cpu.pt). The comparison used the JFK sample on an M1 Mac and mel-spec 0.4.0.

Feature comparison across the full tensor:

The mel-spec benchmark gives 2.26 ms for 0.4.1 and 1.13 ms for 0.5.0 in this configuration. On the first three seconds of the JFK sample, the 0.5.0 features differ from 0.4.1 by a mean absolute difference of 1.2e-6.

GPU Backends

The CPU path is the default. Experimental native GPU backends are available behind feature flags:

Build Checks

The top-level library tests do not automatically build every standalone example crate. Use these commands when changing examples:

cargo test --release
MEL_SPEC_FUZZ_SCALE=50 cargo test --release fuzz_
cargo build --release --manifest-path examples/mel_tga/Cargo.toml
cargo build --release --manifest-path examples/stream_whisper/Cargo.toml
cargo build --release --manifest-path examples/tga_whisper/Cargo.toml
cargo run --release --manifest-path examples/vad_ten_eval/Cargo.toml
(cd examples/browser && npm ci && npm test)

MEL_SPEC_FUZZ_SCALE multiplies the number of cases in the seeded random tests. The Whisper examples compile against the wavey-ai/whisper-rs fork and require a GGML Whisper model to run inference.

Hush Demo

The Hush live browser demo is active at:

https://wavey.ai/code/hush/?v=20261003-36

The source remains at wavey-ai/hush. With the current tuned settings it works as a live browser VAD, spectrogram debugging view, and local Whisper WASM transcription demo. It exposes mel structure, Sobel edges, ridge tracks, candidate speech regions, and the local transcript in real time. The VAD itself should still be treated as experimental rather than a drop-in replacement for a learned VAD. One strong use case is as a browser-side feature/debugging front end or cheap prefilter before a stronger VAD/ASR model.