alyze

CJK analyzers

Enable smartcn (Chinese), nori (Korean), or kuromoji (Japanese) independently, or use cjk to enable all three. All are disabled by default.

[dependencies]
alyze = { version = "0.2.0", features = ["nori"] }

Enabled analyzers are available under alyze::cjk, for example alyze::cjk::nori. Only enabled analyzers and their dictionaries are compiled. Published packages download and verify dictionaries at build time; see offline build instructions.

Use the common analyzer API with any enabled CJK pipeline:

use alyze::analyze::{Analyzer, ReusableBuffer};

let analyzer = Analyzer::nori(Default::default());
let mut buffer = ReusableBuffer::new();
analyzer.analyze("한국어를 분석합니다", &mut buffer, |token| {
    println!("{}: {}", token.position, token.text);
    true
});

Use Analyzer::smartcn, Analyzer::nori, or Analyzer::kuromoji with the corresponding options from alyze::cjk. Reuse the buffer across calls; keep any borrowed user dictionaries or stop lists alive for as long as the analyzer.

Text-match features

The companion alyze-features crate computes query/document text-match signals (e.g. fieldMatch, elementCompleteness, elementSimilarity, textSimilarity) for use in a second-stage re-ranker, mirroring Vespa's rank features so the same signals are available client-side. It's a separate, Apache-2.0 crate (see below) and lives in this workspace.

License

alyze contains MIT-licensed code and Apache-2.0-licensed CJK implementations derived from Apache Lucene. See LICENSE, LICENSE-APACHE, NOTICE, and the CJK documentation for dictionary attribution and licensing.

The separate alyze-features crate is Apache-2.0; its features derive from Vespa, Copyright Vespa.ai.