ctranslate2-rs

• mkl: Enables Intel MKL support
• system-mkl: Enables Intel MKL support using the system-installed MKL library instead of downloading and building it
• openblas: Enables OpenBLAS support (OpenBLAS needs to be installed manually via vcpkg on Windows)
• dnnl: Enables oneDNN support
• ruy: Enables Ruy support
• accelerate: Enables Apple Accelerate support (macOS only)
• openmp-runtime-comp: Enables OpenMP runtime support
• openmp-runtime-intel: Enables OpenMP runtime support for Intel compilers
• disable-cpu-dispatch: Disables CPU kernel dispatching at runtime (should be set when explicitly targeting an architecture)

Multiple features can be enabled at the same time.

To enable Streaming SIMD Extensions 4.1 (SSE4.1), add the -C target-feature=+sse4.1 flag to RUSTFLAGS environment variable.

By default, the ruy feature is enabled.

If you want to use platform-specific default features, use the ct2rs-platform crate.

GPU-Specific Features

• flash-attention: Enables Flash Attention
• tensor-parallel: Enables Tensor Parallelism
• cuda-dynamic-loading: Enables dynamic loading of CUDA libraries at runtime instead of static linking (requires CUDA >= 11)
• cuda-small-binary: Reduces binary size by compressing device code

Tokenizer Features

• sentencepiece: Enables SentencePiece tokenizer support
• tokenizers: Enables HuggingFace's Tokenizers library support
• all-tokenizers: Enables both sentencepiece and tokenizers support

Additional Features

• whisper: Enables Whisper model support
• hub: Enables HuggingFace Hub integration
• system: Skip compiling CTranslate2 and use the system's pre-installed shared library instead (requires setting the appropriate environment variables to locate the CTranslate2 shared library)

Platform Specific Features

When ct2rs-platform is used, the following features are automatically selected based on the platform:

• Windows: openmp-runtime-intel, dnnl, cuda, cudnn, cuda-dynamic-loading, mkl
• Intel MacOS: dnnl, mkl
• Apple Silicon MacOS: accelerate, ruy
• Linux (non-ARM): dnnl, openmp-runtime-comp, cuda, cudnn, cuda-dynamic-loading, mkl, tensor-parallel
• Linux (ARM): openmp-runtime-comp, openblas, ruy

Supported Models

The ct2rs crate has been tested and confirmed to work with the following models:

• BART
• BLOOM
• FALCON
• Marian-MT
• MPT
• NLLB
• GPT-2
• GPT-J
• OPT
• T5
• Whisper

Please see the respective examples for each model.

Stream API

This crate also offers a streaming API that utilizes callback closures. Please refer to the example code for more information.

Model Conversion for CTranslate2

To use model files with CTranslate2, they must first be converted. Below is an example of how to convert the nllb-200-distilled-600M model:

pip install ctranslate2 huggingface_hub torch transformers
ct2-transformers-converter --model facebook/nllb-200-distilled-600M --output_dir nllb-200-distilled-600M \
    --copy_files tokenizer.json

For more details, please refer to the CTranslate2's docs.

Development

When using mise in this repository, sccache is enabled by default for Rust and C/C++ compilation. Run mise install first. CTranslate2 and its dependencies are still built from source; cached compiler outputs can speed up later builds. The first build with an empty cache may be slower.

mise builds use target/sccache to keep their CMake configuration separate from builds without sccache. To run a build without sccache in a POSIX shell, use a separate target directory:

env -u RUSTC_WRAPPER -u CMAKE_C_COMPILER_LAUNCHER -u CMAKE_CXX_COMPILER_LAUNCHER \
  CARGO_TARGET_DIR=target/no-sccache cargo build -p ct2rs --release

Use the same target directory when switching back to cached builds only after reconfiguring CMake; otherwise its cached compiler launcher settings may persist. cargo clean under mise only cleans the active CARGO_TARGET_DIR. These settings are local development defaults and are not required for users of the published crate.

License

This application is released under the MIT License. For details, see the LICENSE file.

CTranslate2 (which this crate builds and links to) is licensed under MIT License. If you redistribute binaries or source that include CTranslate2, you must comply with the MIT terms, including preserving license and notice files. For details, see CTranslate2’s upstream LICENSE/NOTICE.

The published crate may build and/or redistribute artifacts originating from the following third-party projects. Each component remains under its own license, which applies in addition to this crate’s MIT license. When redistributing your software, ensure that you include the required attributions and license texts for any components that were used by your build configuration and target platform. The exact set of components used can depend on enabled features, target architecture, and toolchain.

The published crate also includes the following libraries:

BSD-2-Clause:

• cpuinfo

BSD-3-Clause:

• CUTLASS
• GoogleTest
• cub

Apache License 2.0:

• cpu_features
• The ruy matrix multiplication library
• Thrust: Code at the speed of light

MIT License:

• cxxopts
• spdlog