orp

🧩 ORP: a Lightweight Rust Framework for Building ONNX Runtime Pipelines with ORT

💬 Introduction

orp is a lightweight framework designed to simplify the creation and execution of ONNX Runtime Pipelines in Rust. Built on top of the 🦀 ort runtime and the 🔗 composable crate, it provides an simple way to handle data pre- and post-processing, chain multiple ONNX models together, while encouraging code reuse and clarity.

🔨 Sample Use-Cases

• 🌿 gline-rs: inference engine for GLiNER models
• 🧲 gte-rs: text embedding and re-ranking

⚡️ GPU/NPU Inferences

The execution providers available in ort can be leveraged to perform considerably faster inferences on GPU/NPU hardware.

The first step is to pass the appropriate execution providers in RuntimeParameters. For example:

let rtp = RuntimeParameters::default().with_execution_providers([
    CUDAExecutionProvider::default().build()
]);

The second step is to activate the appropriate features (see related section below), otherwise ir may silently fall-back to CPU. For example:

$ cargo run --features=cuda ...

Please refer to doc/ORT.md for details about execution providers.

📦 Crate Features

This create mirrors the following ort features: