regexr

• Atomic groups (?>...) and possessive quantifiers. Both return a parse error.
• The \N{NAME} named Unicode escape. It returns a parse error.

Quick Start

Add the crate with cargo add regexr. For JIT and SIMD, enable the full feature:

cargo add regexr --features full

use regexr::Regex;

let re = Regex::new(r"\w+").unwrap();
assert!(re.is_match("hello"));

// Find the first match
if let Some(m) = re.find("hello world") {
    println!("Found: {}", m.as_str()); // "hello"
}

// Find all matches
for m in re.find_iter("hello world") {
    println!("{}", m.as_str());
}

Usage

Capture Groups

use regexr::Regex;

let re = Regex::new(r"(\w+)@(\w+)\.(\w+)").unwrap();
let caps = re.captures("user@example.com").unwrap();

println!("{}", &caps[0]); // "user@example.com"
println!("{}", &caps[1]); // "user"

Named Captures

use regexr::Regex;

let re = Regex::new(r"(?P<user>\w+)@(?P<domain>\w+\.\w+)").unwrap();
let caps = re.captures("user@example.com").unwrap();

println!("{}", &caps["user"]);   // "user"
println!("{}", &caps["domain"]); // "example.com"

JIT Compilation

Enable JIT for a pattern that runs many times. JIT needs the jit feature.

use regexr::RegexBuilder;

let re = RegexBuilder::new(r"\w+")
    .jit(true)
    .build()
    .unwrap();

assert!(re.is_match("hello"));

Prefix Optimization

For patterns with many literal alternatives, such as keyword lists in tokenizers:

use regexr::RegexBuilder;

let re = RegexBuilder::new(r"(function|for|while|if|else|return)")
    .optimize_prefixes(true)
    .build()
    .unwrap();

assert!(re.is_match("function"));

Replacement

use regexr::Regex;

let re = Regex::new(r"\d+").unwrap();

assert_eq!(re.replace("abc 123 def", "NUM"), "abc NUM def");
assert_eq!(re.replace_all("abc 123 def 456", "NUM"), "abc NUM def NUM");

Bounded Execution

Backreferences can take exponential time. Set a step budget with RegexBuilder::backtrack_limit:

use regexr::RegexBuilder;

let re = RegexBuilder::new(r"(a+)\1")
    .backtrack_limit(1_000_000)
    .build()
    .unwrap();

match re.try_is_match("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!") {
    Ok(matched) => println!("matched: {matched}"),
    Err(e) => println!("budget exhausted: {e}"),
}

try_find, try_captures, and try_is_match return an error when the budget runs out. find, captures, and is_match do not report that case.

Engine Selection

regexr picks one engine for each pattern. Call Regex::engine_name() to read the choice.

Engines without jit (default simd build):

With the jit feature on x86-64 or aarch64, the matching JIT engines are Jit, JitShiftOr, TaggedNfaJit, and BacktrackingJit.

Capture extraction uses a one-pass engine when the pattern allows it. docs/architecture.md explains the selection rules in full.

Feature Flags

cargo build                                  # simd (default)
cargo build --features full                  # jit + simd
cargo build --no-default-features            # scalar search, no JIT
cargo check --no-default-features --target wasm32-unknown-unknown

Platform Support

• JIT: x86-64 and aarch64. Windows x86-64 uses the Windows calling convention.
• SIMD: AVX2 on x86-64, detected at run time. NEON on aarch64. A scalar fallback runs on every other target.
• WASM: wasm32-unknown-unknown builds with --no-default-features. CI checks this build.

Benchmarks

Run cargo bench --bench competitors to compare regexr with regex, fancy-regex, and pcre2. This README does not publish benchmark numbers. Results depend on the machine.

Speedup relative to the regex crate (higher is better):

Documentation

• docs/architecture.md: engine architecture and selection rules
• docs/features.md: feature reference
• docs/engine_structure.md: engine module layout

Citation

@software{regexr2025,
  author       = {Syah, Farhan},
  title        = {regexr: A Rust Regex Engine with JIT Compilation for LLM Tokenization},
  year         = {2025},
  url          = {https://github.com/ml-rust/regexr},
  note         = {Rust regex engine with lookaround support and JIT compilation}
}

Contributing

Contributions are welcome. See CONTRIBUTING.md for conventions and checks.

Questions and design discussion happen on Discord.

License

Licensed under either of MIT or Apache-2.0, at your option.