hdf5-pure

[!WARNING] Several Rust HDF5 projects claim compatibility and safety while overlooking format requirements that affect correctness.

Avoiding writes and in-place edits does not eliminate the risk: reads can silently return incorrect values. Filters, chunk storage and fill values interact in ways that affect even common datatypes. Numeric layouts and conversion rules introduce further requirements. An implementation can overlook these semantics, or accept content that message flags require it to reject, while passing tests on ordinary files written by libhdf5.

hdf5-pure takes a bottom-up approach: format correctness and compliance with the HDF5 specification come first, with values, metadata and file layouts tested against libhdf5. Development continues on the edit engine, chunk-reading runtime, allocation policies and cache configuration to support more demanding workloads and improve performance. Supported operations and known limitations are documented in the compatibility matrix.

Features

• Read and write HDF5 datasets, groups, attributes and nested hierarchies, with contiguous and chunked dataset storage.
• In-place editing with staged commits, dataset overwrites, attribute updates, object creation and deletion, and copying within or between files. Eligible files support bounded-memory editing.
• Streaming and partial reads with on-demand I/O, custom sources, row windows and multidimensional region selection.
• Appending with immediate, staged and buffered writes to growing datasets.
• File-space management with configurable allocation strategies, persistent free-space reuse and inspection.
• Configurable caching for metadata, decoded chunks and parsed chunk indexes, with memory budgets and statistics.
• Repacking to compact files, omit selected objects, choose output allocation settings and preserve eligible compressed chunks without recompression.
• SWMR for concurrent readers following an append-only writer, interoperable with libhdf5 and h5py. Writing supports unfiltered, chunk-aligned appends to 1-D unlimited datasets.
• MATLAB v7.3 .mat files with serde-based reading and writing of matrices, structs, cell arrays and complex values, alongside lower-level MATLAB convention helpers.
• Typed data with numeric conversions, compound types, enumerations, array types, variable-length strings and complex datasets represented as compound {real, imag} values. Optional ndarray integration provides N-dimensional array I/O.
• Filters including Deflate, shuffle, Fletcher32, LZF, scale-offset and optional ZFP.
• Content verification with SHA-256 hashes and creator metadata through the optional provenance feature.
• No C dependencies with WebAssembly support and lower-level primitives available on bare-metal targets through no_std with alloc.

Examples

Runnable, self-checking examples live in examples/. The table highlights common workflows. See the directory for the complete set. Run any with cargo run --example <name>:

The matlab_fixtures example (run with --features serde) writes .mat v7.3 files for verification in MATLAB/Octave.

Installation

$ cargo add hdf5-pure

Or add it to Cargo.toml by hand:

[dependencies]
hdf5-pure = "0.47.0"

That pulls the default feature set (std, checksum, and deflate), which covers file I/O, the high-level reader and writer API, and deflate compression. The Cargo features section below points to the canonical feature documentation.

For WebAssembly, keep the default features and add the target:

$ cargo build --target wasm32-unknown-unknown

A WASM build uses the in-memory API: FileBuilder::finish serializes the file into a Vec<u8> and File::from_bytes parses one.

Quick start

Writing

use hdf5_pure::{FileBuilder, AttrValue};

let mut builder = FileBuilder::new();

// Datasets
builder.create_dataset("temperature")
    .with_f64_data(&[22.5, 23.1, 21.8])
    .with_shape(&[3]);

// Groups with nested datasets
let mut grp = builder.create_group("sensors");
grp.create_dataset("pressure").with_f32_data(&[101.3, 101.5]);
grp.set_attr("location", AttrValue::AsciiString("lab_a".into()));
builder.add_group(grp.finish());

// Attributes on the root group
builder.set_attr("version", AttrValue::I64(2));

builder.write("output.h5").unwrap();

Reading

use hdf5_pure::File;

let file = File::open("output.h5").unwrap();
let ds = file.dataset("temperature").unwrap();

println!("shape: {:?}", ds.shape().unwrap());    // [3]
println!("data:  {:?}", ds.read_f64().unwrap());  // [22.5, 23.1, 21.8]

let attrs = file.root().attrs().unwrap();
println!("version: {:?}", attrs.get("version"));  // Some(I64(2))

Generic over the element type

DatasetBuilder::with_data(&[T]) and Dataset::read::<T>() let code work over the numeric scalar, fixed byte-array, and FixedAscii string types through the sealed H5Element trait. Numeric reads can lose precision during conversion. Byte arrays and ASCII strings require compatible datatypes and matching widths.

See the generic element I/O guide for supported element types, coercion behavior, and examples.

Editing in place

File::open_rw opens an existing file for staged object edits and dataset appends without rewriting the file from scratch. It selects a bounded or mirrored backing according to the file and access properties.

use hdf5_pure::File;

let file = File::open_rw("output.h5").unwrap();
let root = file.root();

let run2 = root.create_group("run2").unwrap();
run2.create_dataset("signal", |b| { b.with_f64_data(&[1.0, 2.0, 3.0]); }).unwrap();
file.copy("temperature", "temperature_backup").unwrap();
root.delete("sensors/pressure").unwrap();

file.commit().unwrap();

The editing guide covers the backing choice, append modes, durability, locking, space reuse, and format-specific rules. See the compatibility matrix for the exact support status of on-disk structures such as dense storage and chunk indexes.

Reclaiming space (repack)

Deleting objects can make their storage reusable without making the file itself smaller. repack writes the surviving objects into a fresh compact file, optionally dropping paths.

use hdf5_pure::{repack, RepackOptions};

let options = RepackOptions::new()
    .drop_path("scratch")
    .drop_path("runs/aborted");
repack("input.h5", "compact.h5", &options).unwrap();

The repack guide documents the fidelity guarantee, verbatim chunk-copy path, re-encoding fallbacks, reference handling, and the cases that are rejected.

File-space management

FileSpaceSettings combines the allocation strategy, persistence choice, free-space threshold, and checked page size. Apply it with FileBuilder::with_file_space_settings or through FileCreateProperties. Unset settings omit the File Space Info message, while an explicit default value records it. File::file_space_settings reads the recorded configuration and reports malformed metadata as an error.

Persistent paged files support staged edits. Immediate appends also require a file without a userblock. Nonpersistent paged files require mirrored editing, support eligible staged edits, and reject immediate appends.

See the file-space guide for strategy semantics, persisted free-space reuse, paged files, and editing behavior.

Streaming large files

File::open_streaming reads metadata and dataset storage on demand while keeping the same dataset-reading API as File::open.

use hdf5_pure::File;

let file = File::open_streaming("huge.h5").unwrap();
let values = file.dataset("signal").unwrap().read_f64().unwrap();

The streaming guide covers row-window reads, cache budgets, caller-supplied Source implementations, and the bounded-memory read/write paths.

In-memory (WASM)

use hdf5_pure::FileBuilder;

let mut builder = FileBuilder::new();
builder.create_dataset("x").with_f64_data(&[1.0, 2.0]);

let bytes: Vec<u8> = builder.finish().unwrap(); // no filesystem needed

N-dimensional arrays (ndarray feature)

The ndarray feature adds rank-generic dataset I/O through the ndarray crate:

use hdf5_pure::{File, FileBuilder};
use ndarray::{array, Array2};

let a: Array2<f64> = array![[1.0, 2.0], [3.0, 4.0]];
let mut builder = FileBuilder::new();
builder.create_dataset("m").with_ndarray(&a);
let file = File::from_bytes(builder.finish().unwrap()).unwrap();
let back: Array2<f64> = file.dataset("m").unwrap().read_array().unwrap();
assert_eq!(a, back);

See the N-dimensional array guide for feature setup, memory order, dynamic-rank reads, and compression.

SWMR (single writer, multiple readers)

SWMR lets one process append to an unlimited dataset while other processes follow it with refreshing reads. It works with the reference HDF5 C library and h5py.

See the SWMR guide for dataset layout requirements, writer and reader usage, durability, crash recovery, and the exact supported subset. The compatibility matrix records the support status.

Supported data types

Datasets

Compound types

Compound records, complex-number helpers, enumerations, fixed-size arrays, and object references have dedicated builders and readers. See the compound types guide for layouts, tuple helpers, complex values, enumerations, arrays, and references.

Variable-length string reads

Dataset::read_string reads fixed- and variable-length HDF5 string datasets. For large variable-length datasets, the API also supports measuring payload size, applying allocation limits, and visiting values one at a time.

See the variable-length string guide for the bounded and streaming read APIs.

Attributes

Compression

Dataset filters include deflate, shuffle, Fletcher32, LZF, scale-offset, and optional ZFP. Filters operate on chunked storage. Reads decode the configured pipeline automatically.

# use hdf5_pure::DeflateLevel;
builder.create_dataset("compressed")
    .with_f64_data(&data)
    .with_chunks(&[100])
    .with_shuffle()
    .with_deflate(DeflateLevel::Six);

See the compression guide for filter semantics, chaining rules, scale-offset and ZFP behavior, feature flags, and interoperability.

Userblock (MATLAB v7.3)

use hdf5_pure::{FileBuilder, UserblockSize};
use hdf5_pure::mat::userblock;

let mut builder = FileBuilder::new();
builder.with_userblock(UserblockSize::MIN);
builder.with_userblock_content(&userblock::header_block(userblock::DEFAULT_DESCRIPTION));
builder.create_dataset("data").with_f64_data(&[1.0]);

let bytes = builder.finish().unwrap();

MATLAB struct pattern

use hdf5_pure::{FileBuilder, AttrValue};

let mut builder = FileBuilder::new();
let mut grp = builder.create_group("my_struct");

let mut fields = Vec::new();
for (name, data) in [("x", vec![1.0, 2.0]), ("y", vec![3.0, 4.0])] {
    fields.push(name.to_string());
    grp.create_dataset(name).with_f64_data(&data)
        .set_attr("MATLAB_class", AttrValue::AsciiString("double".into()));
}

grp.set_attr("MATLAB_class", AttrValue::AsciiString("struct".into()));
grp.set_attr("MATLAB_fields", AttrValue::VarLenAsciiCharArray(fields));
builder.add_group(grp.finish());

MATLAB v7.3 .mat via serde

With the serde feature, Rust structs can be serialized directly to .mat v7.3 files and back:

These files are written in the HDF5 1.8 on-disk format, which every MATLAB release can open. A version 3 superblock is an HDF5 1.10 addition, and MATLAB used HDF5 1.8.12 before R2021b, so the newer format is unreadable there. Set mat::Options::libver to LibVer::V110 to opt into it, which compression requires. See the mat module.

use hdf5_pure::mat::{self, Complex64, Matrix};
use serde::{Serialize, Deserialize};

#[derive(Serialize, Deserialize, PartialEq, Debug)]
struct Experiment {
    name: String,
    trial: u32,
    samples: Vec<f64>,
    data: Matrix<f64>,
    waveform: Vec<Complex64>,
    config: Config,
}

#[derive(Serialize, Deserialize, PartialEq, Debug)]
struct Config { threshold: f64, tag: String }

let e = Experiment {
    name: "run1".into(), trial: 3,
    samples: vec![1.0, 2.0, 3.0],
    data: Matrix::from_row_major(2, 3, vec![1.0, 2.0, 3.0, 4.0, 5.0, 6.0]),
    waveform: vec![Complex64::new(1.0, 0.0), Complex64::new(0.0, 1.0)],
    config: Config { threshold: 0.5, tag: "prod".into() },
};

mat::to_file(&e, "experiment.mat").unwrap();
let back: Experiment = mat::from_file("experiment.mat").unwrap();
assert_eq!(back, e);

The top-level value must be a struct (or HashMap<String, _>); each field becomes a MATLAB variable. Mapping:

Cell array pattern

Sequences that don't unify into a numeric matrix lower to a MATLAB cell array. Each element is interned under the conventional #refs# group and the parent dataset stores object references.

use hdf5_pure::mat;
use serde::{Serialize, Deserialize};

#[derive(Serialize, Deserialize)]
struct Point { x: f64, y: f64 }

#[derive(Serialize, Deserialize)]
struct Capture {
    /// 3x1 cell array of struct.
    path: Vec<Point>,
    /// 3x1 cell array; the `None` slot becomes `struct([])`.
    optionals: Vec<Option<Point>>,
    /// Outer 2x1 cell of cells; rows-of-variable-length-records shape.
    grid: Vec<Vec<Option<Point>>>,
    /// Ragged numerics also fall back to cell rather than erroring.
    ragged: Vec<Vec<f64>>,
}

In MATLAB this loads as iscell(path) == true, path{1}.x, etc. Empty None slots load as struct([]) (isempty(fieldnames(...))).

Reader compatibility. Cell arrays load correctly in MATLAB, libmatio (reference C library), Julia's MAT.jl, and Python via pymatreader / hdf5storage. GNU Octave 11's load does not yet follow object references for v7.3 cells (warns "unknown datatype"); load such files with one of the above instead.

Reading provides MatDatetime, MatDuration, MatCategorical, MatEnum, MatTable and MatTimetable views. MATLAB containers.Map values deserialize into Rust maps. Other opaque classes retain their raw property maps.

Not supported for writing: non-unit enum variants, MATLAB objects (classdef), datetime / duration / categorical types.

Cargo features

The default feature set is std, checksum, and deflate. Optional features are fast-deflate, provenance, zfp, ndarray, serde, and num-complex. The Cargo features section of the crate documentation is the canonical description of what each feature enables and which dependencies or other features it implies.

For bare-metal no_std, disable default features and keep checksum for metadata validation:

[dependencies]
hdf5-pure = { version = "0.47.0", default-features = false, features = ["checksum"] }

The high-level File / FileBuilder API is std-gated, so a no_std build exposes only the lower-level primitives.

Limitations

Unsupported and partially supported HDF5 file-format features are listed in the HDF5 compatibility matrix. Operations that cannot preserve data faithfully return a typed error.

Contributing

See CONTRIBUTING.md. The AI policy and the code style apply to every contribution.

History

hdf5-pure was started by Stephen Berry, as a derived project of rustyhdf5 which was a ~25k LoC slop implementation of HDF5 with wild claims about performance over libhdf5. rustyhdf5 of course didn't support 98% of HDF5 features and was extremely buggy.

Under Stephen, hdf5-pure rapidly developed into an ambitious and respectable project, which read and wrote most HDF5 files without issues. Stephen eventually transferred the project ownership to Marc König who'd been aiding development with numerous bug reports and general recommendations to improvements. Marc König is a long time HDF5 user and contributor to hdf5-rust (hdf5-metno) and hdf5-pure development continued under him.

License

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)

at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Third-party code

• The scale-offset filter in crates/hdf5-pure-filter/src/scaleoffset/mod.rs is a port of H5Zscaleoffset.c from the HDF5 library, which The HDF Group licenses under the BSD 3-Clause license in LICENSE-HDF5. The crosscheck tests build that library to compare against, and nothing of it ships in the crate.
• The ZFP codec in crates/hdf5-pure-filter/src/zfp.rs is a port of the reference implementation by Lawrence Livermore National Security, LLC, under the BSD 3-Clause license in LICENSE-ZFP.
• The metadata checksum in crates/hdf5-pure-format/src/checksum.rs is Bob Jenkins' public-domain lookup3 hash.
• The HDF5 format parsing and low-level I/O modules are derived from rustyhdf5 by the RustyStack project, MIT licensed.