hibiki

How is it trained?

Hibiki relies on supervised training of aligned source speech and target speech and text, from the same speaker. Such data does not exist in significant amounts so we rely on synthetic data generation. Word-level matching is made between source and target transcripts using a contextual alignment weakly-supervised method that leverages an off-the-shelf MADLAD machine translation system. The derived alignment rule (a word should only appear in the target once it's predictable from the source) is applied either by inserting silences or by synthesizing targets with a voice controlled, alignment-aware TTS.

Inference

At inference, Hibiki continuously encodes source speech and produces target speech. Hibiki relies on simple temperature sampling and is thus compatible with batching unlike models that rely on complex inference policies. Moreover, the fidelity of Hibiki's voice transfer can be controlled by changing the coefficient of the Classifier-Free Guidance: a larger coefficient will increase voice similarity, but excessive coefficients can lead to worse translations. Hibiki currently only supports French-to-English translation. Its smaller alternative, Hibiki-M can run locally on smartphone hardware. Current models were trained on sequences up to 120 seconds and use a context size of 40 seconds.

Running the model

We provide inference code for PyTorch, Rust, MLX for macOS, and MLX-swift for iOS. Note that the implementation for Hibiki is very close to that of Moshi, and the actual code is in the kyutai-labs/moshi repository.

PyTorch

In order to translate an audio file using Hibiki/PyTorch, install the moshi package via.

pip install -U moshi

Then you can retrieve some sample files from kyutai-labs/moshi and translate them via the following:

wget https://github.com/kyutai-labs/moshi/raw/refs/heads/main/data/sample_fr_hibiki_crepes.mp3
python -m moshi.run_inference sample_fr_hibiki_crepes.mp3 out_en.wav --hf-repo kyutai/hibiki-1b-pytorch-bf16

You can specify some classifier-free guidance using the --cfg-coef parameter. The default value is 1, the higher the value, the closer the generated voice should be to the original voice. A typical value to use is 3.

MLX

In order to translate an audio file using Hibiki/MLX, install the moshi_mlx package via the following command. You need at least version 0.2.1 of this package.

pip install -U moshi_mlx

Then you can retrieve some sample files from kyutai-labs/moshi and translate them via the following:

wget https://github.com/kyutai-labs/moshi/raw/refs/heads/main/data/sample_fr_hibiki_crepes.mp3
python -m moshi_mlx.run_inference sample_fr_hibiki_crepes.mp3 out_en.wav --hf-repo kyutai/hibiki-1b-mlx-bf16

You can specify some classifier-free guidance using the --cfg-coef parameter. The default value is 1, the higher the value, the closer the generated voice should be to the original voice. A typical value to use is 3.

You can also use the model in real-time via the web-ui by running the following command.

python -m moshi_mlx.local_web --hf-repo kyutai/hibiki-1b-mlx-bf16

MLX-Swift

The kyutai-labs/moshi-swift repo contains a MLX-Swift implementation that can run on an iPhone. This was tested on an iPhone 16 Pro. Note that this code there is very much experimental.

Rust

The hibiki-rs directory contains a simple Rust app that generates a translation based on an original audio file. You can use either --features cuda to run on a nvidia gpu, or --features metal to run on a mac.

cd hibiki-rs
wget https://github.com/kyutai-labs/moshi/raw/refs/heads/main/data/sample_fr_hibiki_crepes.mp3
cargo run  --features metal -r -- gen sample_fr_hibiki_crepes.mp3 out_en.wav

Models

We release two models for FR -> EN translation:

• Hibiki 2B (for the backbone, a bit more with the depth transformer), 16 RVQ per stream.
• Hibiki 1B (for the backbone, a bit more with the depth transformer), 8 RVQ per stream, ideal for on device inferencde.

Depending on the backend, the file format will vary. Quantized models coming up soon. Current list of models:

• Hibiki 2B for PyTorch (bf16): kyutai/hibiki-2b-pytorch-bf16
• Hibiki 1B for PyTorch (bf16): kyutai/hibiki-1b-pytorch-bf16
• Hibiki 2B for MLX (bf16): kyutai/hibiki-2b-mlx-bf16
• Hibiki 1B for MLX (bf16): kyutai/hibiki-1b-mlx-bf16

All models are released under the CC-BY 4.0 license.

License

The present code is provided under the MIT license for the Python parts, and Apache license for the Rust backend. The web client code is provided under the MIT license.

The weights for the models are released under the CC-BY 4.0 license.

Citation

If you use Hibiki, please cite the following paper,

@misc{kyutai2025hibiki,
      title={High-Fidelity Simultaneous Speech-To-Speech Translation},
      author={Tom Labiausse and Laurent Mazar\'e and Edouard Grave and
      Patrick P\'erez and Alexandre D\'efossez and Neil Zeghidour},
      year={2025},
      eprint={2502.03382},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.03382},
}