stringcheese-linkage

Probabilistic record linkage — the Fellegi-Sunter (1969) classifier and sorted-neighborhood blocking (Hernández & Stolfo, 1995) — built on top of the StringCheese string-processing toolkit.

StringCheese owns string processing: comparing, manipulating, segmenting, normalizing, phonetically encoding, and indexing sequences — given two strings, produce a distance, similarity, alignment, phonetic key, transformed string, or match result. This crate is the record-linkage layer above it: given two whole records (each with multiple fields), decide whether they refer to the same real-world entity by combining StringCheese's per-field similarities under a statistical model.

Scope

• Fellegi-Sunter classifier — the log-likelihood combination of per-field agreements into a Match / NonMatch / PossibleMatch decision, with two-threshold classification per the original 1969 formulation.
• Parameter estimation — labelled-pairs MLE (with Jeffreys smoothing), and a reserved API surface for an EM estimator (currently stubbed as NotYetImplemented).
• Blocking — sorted-neighborhood blocking as a candidate-generation helper, so pairwise scoring at scale doesn't need to be O(n²).

Not in scope

• String processing itself — that's StringCheese. This crate consumes per-field similarity scores; how you compute them (Levenshtein, Jaro-Winkler, phonetic-key equality, whatever) belongs to stringcheese-compare and its siblings.
• Learned similarity models — see StringCheese's design notes; this crate might grow a LearnedFieldSimilarity trait as an integration point, but ships no models.
• Storage / database integration — record I/O is up to the caller. This crate operates on in-memory &[f64] per-field score vectors.

Layout

src/
├── lib.rs
├── model.rs           # LinkageModel and its threshold-based classifier
├── classifier.rs      # LinkageDecision enum
├── field.rs           # FieldComparator + FieldStrategy + AgreementRule
├── weight.rs          # agree_weight / disagree_weight log-likelihood
├── estimation.rs      # PriorProbabilities, LabeledPairsEstimator, EmEstimator (stub)
├── error.rs           # LinkageModelError, LinkageScoreError, EstimationError
├── golden.rs          # golden test fixtures (test-only)
├── property_tests.rs  # proptest properties (test-only)
└── blocking/
    ├── mod.rs
    └── sorted_neighborhood.rs   # Hernández-Stolfo 1995 blocking

Building

Path deps to StringCheese assume the two repos are siblings:

git/
├── stringcheese/            # StringCheese umbrella
└── stringcheese-linkage/    # this repo

If your layout differs, edit the path = "../stringcheese/…" entries in Cargo.toml. When StringCheese publishes to crates.io, these will flip to version deps.

Build and test with the usual cargo commands:

cargo build --all-features
cargo test  --all-features
cargo clippy --all-features --all-targets -- -W clippy::pedantic
cargo fmt --check

License

Dual MIT / Apache-2.0, matching StringCheese.

References

• Fellegi, I. P., & Sunter, A. B. (1969). "A theory for record linkage." Journal of the American Statistical Association, 64(328), 1183–1210. DOI: https://doi.org/10.1080/01621459.1969.10501049
• Hernández, M. A., & Stolfo, S. J. (1995). "The merge/purge problem for large databases." Proceedings of the 1995 ACM SIGMOD International Conference on Management of Data, 127–138. DOI: https://doi.org/10.1145/223784.223807
• Christen, P. (2012). Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection. Springer. ISBN 978-3-642-31163-5.