xet-core

Rust Crates (crates.io)

Python Package (PyPI)

Built from the hf_xet/ directory using maturin.

CLI Binary

Built from the git_xet/ directory. Distributed via GitHub releases.

Contributions (feature requests, bugs, etc.) are encouraged & appreciated 💙💚💛💜🧡❤️

Please join us in making xet-core better. We value everyone's contributions. Code is not the only way to help. Answering questions, helping each other, improving documentation, filing issues all help immensely. If you are interested in contributing (please do!), check out the contribution guide for this repository.

Issues, Diagnostics & Debugging

If you encounter an issue with hf-xet, please collect diagnostic information and attach it when creating a new Issue.

The scripts/diag/ directory contains platform-specific scripts that download debug symbols, configure logging, and capture periodic stack traces and core dumps:

# prefix your failing command with the script for your OS, e.g.:
./scripts/diag/hf-xet-diag-macos.sh -- python my-script.py

See scripts/diag/README.md for full usage, output layout, dump analysis instructions, and how to install debug symbols manually.

Quick debugging environment variables:

RUST_BACKTRACE=full          # full Rust backtraces on panic
RUST_LOG=info                # enable hf-xet logging
HF_XET_LOG_DEST=/tmp/xet.log # log file, or a directory with a trailing / (defaults to ~/.cache/huggingface/xet/logs/)

Telemetry

hf-xet reports a small performance summary — byte counts, dedup effectiveness, throughput, and wall time — to the CAS server at the end of each upload or download. It is best-effort: never retried, and it cannot delay or fail a transfer. The payload contains no file names, paths, hashes, repository ids, or user ids.

To turn it off:

HF_XET_TELEMETRY_ENABLED=0

It is also off whenever huggingface_hub's telemetry opt-outs are set: HF_HUB_DISABLE_TELEMETRY, DISABLE_TELEMETRY, DO_NOT_TRACK, HF_HUB_OFFLINE, or TRANSFORMERS_OFFLINE (any of 1, ON, YES, TRUE). These win over HF_XET_TELEMETRY_ENABLED=1.

Tuning (rarely needed):

HF_XET_TELEMETRY_FINAL_FLUSH_TIMEOUT=2s  # how long a transfer waits for its final report; 0 = don't wait
HF_XET_TELEMETRY_HEARTBEAT_AFTER=5m      # long transfers report progress after this; 0 disables
HF_XET_TELEMETRY_HEARTBEAT_INTERVAL=5m   # and then at this interval
HF_XET_TELEMETRY_REQUEST_TIMEOUT=5s      # per-request budget
HF_XET_TELEMETRY_MAX_IN_FLIGHT=4         # concurrent reports before dropping

Local Development

Repo Organization

• xet_pkg/ (hf-xet): High-level session API for uploading and downloading files with deduplication.
• xet_client/ (xet-client): HTTP client for CAS and Hub backend services.
• xet_data/ (xet-data): Chunking, deduplication, and file reconstruction pipeline.
• xet_core_structures/ (xet-core-structures): MerkleHash, metadata shards, Xorb objects, and shared data structures.
• xet_runtime/ (xet-runtime): Async runtime, configuration, logging, and utilities.
• hf_xet/: Python bindings (maturin/PyO3), produces the hf-xet PyPI package.
• git_xet/: Git LFS compatible CLI tool (git-xet).
• wasm/: WebAssembly builds. hf_xet_thin_wasm is the published thin-client wasm crate; hf_xet_wasm is an example / smoke-test wrapper around xet_pkg exposing both upload and download JS APIs for CI and manual browser testing (not a published SDK).
• simulation/: Simulation and benchmarking infrastructure.

Build, Test & Benchmark

To build xet-core, look at requirements in GitHub Actions CI Workflow for the Rust toolchain to install. Follow Rust documentation for installing rustup and that version of the toolchain. Use the following steps for building, testing, benchmarking.

Many of us on the team use VSCode, so we have checked in some settings in the .vscode directory. Install the rust-analyzer extension.

Build:

cargo build

Test:

cargo test

Benchmark:

cargo bench

Linting:

cargo clippy -r --verbose -- -D warnings

Formatting (requires nightly toolchain):

cargo +nightly fmt --manifest-path ./Cargo.toml --all

Building Python package and running locally (on *nix systems):

1. Create Python3 virtualenv: python3 -mvenv ~/venv
2. Activate virtualenv: source ~/venv/bin/activate
3. Install maturin: pip3 install maturin ipython
4. Go to hf_xet crate: cd hf_xet
5. Build: maturin develop
6. Test:

ipython
import hf_xet as hfxet
hfxet.upload_files()
hfxet.download_files()

Developing with tokio console

Prerequisite is installing tokio-console (cargo install tokio-console). See https://github.com/tokio-rs/console

To use tokio-console with hf-xet there are compile hf_xet with the following command:

RUSTFLAGS="--cfg tokio_unstable" maturin develop -r --features tokio-console

Then while hf_xet is running (via a hf cli command or huggingface_hub python code), tokio-console will be able to connect.

Ex.

# In one terminal:
pip install huggingface_hub
RUSTFLAGS="--cfg tokio_unstable" maturin develop -r --features tokio-console
hf download openai/gpt-oss-20b

# In another terminal
cargo install tokio-console
tokio-console

Building universal whl for MacOS:

From hf_xet directory:

MACOSX_DEPLOYMENT_TARGET=10.9 maturin build --release --target universal2-apple-darwin --features openssl_vendored

Note: You may need to install x86_64: rustup target add x86_64-apple-darwin

Testing

Unit-tests are run with cargo test, benchmarks are run with cargo bench. Some crates have a main.rs that can be run for manual testing.

WebAssembly compatibility

xet_pkg (hf-xet), xet_client, xet_data, xet_core_structures, and xet_runtime must compile cleanly for wasm32-unknown-unknown so that downstream browser consumers (e.g. hf-hub) can depend on this crate. CI enforces this via cargo +nightly check --target wasm32-unknown-unknown -p hf-xet plus the wasm-pack-style builds under wasm/.

The example wrapper wasm/hf_xet_wasm/ is not a published browser SDK — it is a CI smoke test and hand-runnable demo. Real browser consumers should depend on hf-xet directly with their own #[wasm_bindgen] glue, or use a downstream SDK such as hf-hub.

When adding or modifying code in the wasm-reachable crates, please keep the wasm build green:

• Prefer web_time::Instant over std::time::Instant / tokio::time::Instant on code paths reachable from wasm (the std and tokio variants panic on wasm32).
• Use wasm_bindgen_futures::spawn_local via the tokio_with_wasm::alias as tokio shim instead of bare tokio::spawn / JoinSet::spawn. The browser's reqwest backend produces !Send futures.
• Apply the conditional ?Send pattern to #[async_trait] definitions whose methods touch HTTP / I/O:

  #[cfg_attr(not(target_family = "wasm"), async_trait::async_trait)]
  #[cfg_attr(target_family = "wasm", async_trait::async_trait(?Send))]
  

• Gate filesystem-only code (std::fs, tokio::fs, OpenOptions, spawn_blocking, disk caches, file-path download methods) with #[cfg(not(target_family = "wasm"))].

See api_changes/update_260515_wasm_target_support.md for the full set of wasm-only differences and the patterns wasm-reachable code must follow. The example browser pages live at wasm/hf_xet_wasm/examples/download.html and wasm/hf_xet_wasm/examples/upload.html.

References & History

• Technical Blog posts
• Git is for Data 'CIDR paper
• History: xet-core is adapted from xet-core, which contains deep git integration, along with very different backend services implementation.