oxen

🚀 Built for speed

One of the main reasons datasets are hard to maintain is the pure performance of indexing the data and transferring the data over the network. We wanted to be able to index hundreds of thousands of images, videos, audio files, and text files in seconds.

Watch below as we version hundreds of thousands of images in seconds 🔥

But speed is only the beginning.

✅ Features

Oxen is built around ergonomics, ease of use, and it is easy to learn. If you know how to use git, you know how to use Oxen.

• 🔥 Fast (efficient indexing and syncing of data)
• 🧠 Easy to learn (same commands as git)
• 💪 Handles large files (images, videos, audio, text, parquet, json, model weights, csv, etc)
• 🗄️ Index lots of files (millions of images? no problem)
• 📊 Native tabular data handling (index and query csv, parquet, jsonl files)
• 📈 Tracks changes over time (never worry about losing the state of your data)
• 🤝 Collaborate with your team (sync to an oxen-server)
• 🌎 Workspaces to interact with the data on the centralized server
• 👀 Better data visualization on OxenHub

🐮 Learn The Basics

To learn what everything Oxen can do, the full documentation can be found at https://docs.oxen.ai.

🧑‍💻 Getting Started

You can install through homebrew or pip or from our releases page. When upgrading the CLI or a self-hosted oxen-server, check the migrations for any step the new release asks you to run.

🖥️ Platform Support

oxen-server is not supported on Windows — run it on Linux, macOS, or in Docker. The oxen CLI and Python client are supported on Windows and connect to a server hosted on a supported platform.

🐂 Install Command Line Tool

Install via Homebrew:

brew install oxen

🐍 Install Python Library

pip install oxenai

⬇️ Clone Dataset

Clone your first Oxen repository from the OxenHub.

oxen clone https://hub.oxen.ai/ox/CatDogBBox

🤝 Support

If you have any questions, comments, suggestions, or just want to get in contact with the team, feel free to email us at hello@oxen.ai

👥 Contributing

This repository contains the Python library that wraps the core Rust codebase. We would love help extending out the python interfaces, the documentation, or the core rust library.

Code bases to contribute to:

• 🦀 Rust Library & Binaries
• 🐍 Python Interface
• 📚 Documentation

See the CONTRIBUTING.md document for guidelines on external contributions.

All contributors are required to follow the CODE_OF_CONDUCT.md. The tl;dr is be nice and respectful during all engagements :)

If you are building anything with Oxen.ai or have any questions we would love to hear from you in our discord.

Build 🔨

Each codebase has its own build instructions, please refer to the Rust build instructions and oxen-python's build instructions for specifics.

However, each codebase shares the same pre-requisites and pre-commit hooks.

Prerequisites

Automatic Install

You should use bin/install-prereqs to automatically install the required development tools and toolchains for Rust and Python. Execute that as:

bin/install-prereqs

It supports MacOS and Debian-based Linux distributions. If you have a different OS or distribution, or if you have some error with the install script, you can follow the manual installation steps below.

Manual Installation

Oxen is purely written in Rust 🦀. You should install the Rust toolchain with rustup.

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Once you have rust, install the following developer tools:

• bacon: run the server with reload-on-changes
• cargo-machete: identify and remove unused dependencies
• cargo-llvm-cov: calculate test code coverage
• cargo-sort: ensure Cargo.toml files are organized

You can install all of these at once with the following command:

cargo install bacon cargo-machete cargo-llvm-cov cargo-sort

Make sure cmake is installed. cmake can be installed on macOS with:

brew install cmake

The Python interface uses liboxen bindings provided by PyO3.

The oxen-python codebase requires installing uv. Use the version pinned in tool-versions.env:

source ./tool-versions.env && curl -LsSf "https://astral.sh/uv/${UV_VERSION}/install.sh" | sh

If you use mise to manage your Python installs, you may run into an error where the oxen-py crate can't find the Python dynamic library to link with, e.g., dyld[31558]: Library not loaded: @rpath/libpython3.13.dylib

You can fix it by adding this to your mise config (~/.config/mise/config.toml)

[env]
DYLD_LIBRARY_PATH = "{{ exec(command='mise where python') }}/lib"

Pre-Commit Hooks

We use pre-commit-hooks to check for commit consistency.

Install with uv as a tool:

uv tool install pre-commit

Install Oxen's pre-commit hooks locally using:

pre-commit install

Production Release Build

For deployment, build with the production feature flag and --release:

cargo build --workspace --release --features production

This enables:

• OpenTelemetry tracing (otel) -- export spans to any OTLP-compatible collector (Jaeger, Tempo, Datadog, etc.). See OpenTelemetry Tracing for runtime configuration.
• FFmpeg thumbnails (ffmpeg) -- generate video/image thumbnails via FFmpeg (requires FFmpeg libraries installed on the host).
• Performance logging (perf-logging) -- additional timing instrumentation for internal operations.

Without --features production, the default build excludes OTel dependencies and FFmpeg support, keeping the binary smaller for local development.

The published Docker image names liboxen/ffmpeg,oxen-server/otel rather than production, so it ships thumbnails and tracing without the extra timing instrumentation perf-logging adds. Tracing is inert there until an OTLP endpoint is configured at runtime.

Logging

Oxen uses structured logging via the tracing crate. All log output goes to stderr by default in a human-readable format. This applies to the CLI (oxen), the server (oxen-server), and any code using liboxen (including the Python bindings).

Controlling Log Level

Set the RUST_LOG environment variable to control verbosity.

# Show debug logs from the oxen library
RUST_LOG=debug oxen push origin main

# Show only warnings and errors
RUST_LOG=warn oxen-server start

# Fine-grained: debug for liboxen, warn for everything else
RUST_LOG=warn,liboxen=debug oxen-server start

RUST_LOG gates the log destinations — stderr, the JSON file, and error reporting. It does not gate OpenTelemetry span export, which has its own filter (OXEN_OTEL_FILTER); raising log verbosity for a debugging session leaves the exported traces unchanged.

File Logging with OXEN_LOG_DIR

Set OXEN_LOG_DIR to enable file-based logging in addition to stderr. This env var is a directory where rotating log files are written. Log files are written as newline-delimited JSON (one JSON object per line), rotated daily. Each line includes the timestamp, level, target, thread ID, source file, and line number.

OXEN_LOG_DIR=./logs/ RUST_LOG=warn oxen clone https://hub.oxen.ai/ox/CatDogBBox
OXEN_LOG_DIR=/var/log/oxen oxen-server start

Log files are named {app_name}.{date} (e.g. oxen-server.2026-04-06) inside the configured directory.

To ingest these logs with standard tooling:

• Promtail / Grafana Loki -- point a file_sd or static target at the log directory; Loki handles newline-delimited JSON natively.
• Filebeat / Elasticsearch -- configure a filebeat.inputs entry with type: filestream and parsers: [{ ndjson: {} }].
• Vector -- use a file source with decoding.codec = "json".
• jq -- for ad-hoc inspection:

# Stream logs, filter for errors
tail -f ~/.oxen/logs/oxen-server.2026-04-06 | jq 'select(.level == "ERROR")'

Prometheus Metrics

oxen-server can expose a Prometheus-compatible metrics endpoint. Requires the metrics compile-time feature (included in production) and OXEN_METRICS_PORT at runtime. See Prometheus Metrics for details.

OpenTelemetry Tracing

oxen-server can export tracing spans to any OTLP-compatible collector (Jaeger, Tempo, etc.), continuing a trace begun by whoever called it. The release image is built with the otel feature; a local build needs it named explicitly. Export stays off until the standard OTEL_EXPORTER_OTLP_ENDPOINT is set. The CLI and the Python bindings can export too. See OpenTelemetry Tracing for the full env-var reference.

FmtSpan Events

Span lifecycle events can be emitted as log lines on stderr for lightweight tracing. See FmtSpan Events for details.

Why build Oxen?

Oxen was built by a team of machine learning engineers, who have spent countless hours in their careers managing datasets. We have used many different tools, but none of them were as easy to use and as ergonomic as we would like.

If you have ever tried Git LFS to version large datasets and became frustrated, we feel your pain. Solutions like Git LFS are unwieldy and painfully slow in practice. We needed a faster alternative designed for large datasets.

If you have ever uploaded a large dataset of images, audio, video, or text to a cloud storage bucket with the name:

s3://data/images_july_2022_final_2_no_really_final.tar.gz

We built Oxen to be the tool we wish we had.

Why the name Oxen?

"Oxen" 🐂 comes from the fact that the tooling will plow, maintain, and version your data like a good farmer tends to their fields 🌾. Let Oxen take care of the grunt work of your infrastructure so you can focus on the higher-level problems that matter to your product.

License

Oxen is licensed under the Apache License 2.0.