Skip to content

Add Ribbon filter implementation - #54

Merged
thomasmueller merged 5 commits into
FastFilter:masterfrom
amikumar91:feature/ribbon-filter
Jul 30, 2026
Merged

Add Ribbon filter implementation#54
thomasmueller merged 5 commits into
FastFilter:masterfrom
amikumar91:feature/ribbon-filter

Conversation

@amikumar91

@amikumar91 amikumar91 commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Add Ribbon filter implementation

What

Adds a standard (non-homogeneous) Ribbon filter, org.fastfilter.ribbon.RibbonFilter, implementing the Filter interface and registered as FilterType.RIBBON / TestFilterType.RIBBON.

Closes #33

Why

A Ribbon filter is a retrieval data structure: for every key x in the set, solving a banded linear system over GF(2) yields a small function f(x) returning an r-bit fingerprint. It's structurally the closest existing analog in this repo to Xor8/XorBinaryFuse8 — hash a key, build with retries, query by re-deriving the hash and recombining — but the combine step is a dot product against a Gaussian-eliminated band rather than an XOR of three fingerprint slots.

References:

  • Dietzfelbinger & Walzer, Efficient Gauss Elimination for Near-Quadratic Matrices with One Short Random Block per Row, ESA 2019
  • Dillinger & Walzer, Ribbon filter: practically smaller than Bloom and Xor, arXiv:2103.02515

Scope of this PR

Fixed ribbon width w = 64 (single long, no 128-bit/SIMD variant), firstCoeffAlwaysOne = true, row-major byte[] solution storage (one value per slot, mirroring Xor8's byte[] fingerprints), configurable resultBits (default 8, capped at 8 by the byte-per-slot layout). Construction retries with a new seed on linear-dependence failure, and grows the slot-count overhead factor if all seed attempts at a size fail — the same retry shape as Xor8's constructor loop, just with an extra outer loop for slot growth.

Out of scope, called out here as separately-reviewable follow-ups:

  • Homogeneous Ribbon variant (zero-result-vector construction that can't fail)
  • w = 128 / SIMD-friendly widths
  • Interleaved Column-Major Layout (ICML) — the cache-optimized query layout. The current query is an intentionally simple O(w · r) per-column loop; the benchmark below shows this is the main cost relative to the Xor filters, and ICML is the natural fix.
  • RocksDB-style empirical overhead lookup tables (this PR uses a simple numStarts = ceil(size * (1 + overheadFactor)) formula, growing overheadFactor by 1.5x on repeated failure)
  • "Smash" / bumping optimizations

Testing

  • RibbonFilterTest: zero false negatives at sizes 10, 1,000, and 1,000,000; false-positive rate assertion (< 0.01, generous margin around the expected 1/256 ≈ 0.0039) at 1,000,000 keys.
  • Added a RibbonFilter row to the existing parameterized xor/SerializationTest (round-trip via both ByteBuffer and stream, buffer-too-small handling, multi-round serialization, etc.) rather than a new test file, since its constructor/deserializer signatures already fit that harness.
  • mvn test passes locally: 129 tests, 0 failures, 0 errors — including the full pre-existing suite (RegressionTests, TestAllFilters, SmallSetTest, StringFilters, etc.), confirming no regressions to other filter types.

Quick local comparison (1,000,000 keys, setting=10 where applicable):

Filter bits/key fpp construct ns/key lookup ns/key
RIBBON 8.80 0.0056 182 1875
XOR_16 19.68 0.000017 254 18
XOR_BINARY_FUSE_8 9.04 0.0040 114 27

Space is competitive with XorBinaryFuse8 at a similar target fpp, and construction time is in the same ballpark. Lookup is ~70-100x slower than the Xor filters — expected, since this PR intentionally uses the naive per-column dot product instead of ICML; that's the first candidate for a follow-up perf PR.

Checklist

  • RibbonFilter.java implements Filter: banding (on-the-fly Gaussian elimination), back-substitution, query
  • FilterType/TestFilterType updated with RIBBON
  • RibbonFilterTest: zero false negatives + fpp in range, multiple sizes
  • Serialization round-trip coverage (ByteBuffer + stream)
  • mvn test passes locally (129/129)
  • Class-level Javadoc citing the two papers above

Implements the standard (non-homogeneous, w=64) Ribbon filter
  construction and membership query described in issue FastFilter#33, following
  Xor8's structure (hash-derive, build-with-retries, query-by-recombine).
Mirrors Xor8's ByteBuffer/OutputStream serialize + static deserialize
  convention. Unlike Xor8, numSlots/numStarts must be stored explicitly
  (not re-derived from size) since the construction retry loop can grow
  the slot count beyond the simple sizing formula.
Setting is ignored the same way XOR_8/XOR_16/XOR_PLUS_8 ignore it here:
  TestAllFilters.test() hardcodes setting=10 for every TestFilterType,
  which would exceed RibbonFilter's 8-bit (byte-storage) ceiling if
  passed through.
@thomasmueller

Copy link
Copy Markdown
Contributor

It looks good to me! Given that this is a slow implementation (queries are very slow), what about using e.g. RibbonNaiveFilter (RIBBON_NAIVE) or RibbonSlowFilter or something similar? That way, the implementation can be retained later (one a faster implementation is available), which I think would make sense to better understand the algorithm.

@thomasmueller

Copy link
Copy Markdown
Contributor

Class-level Javadoc citing the two papers above

Yes I think that would make sense!

Reserves the RibbonFilter/RIBBON name for a future
  ICML-backed implementation with fast queries; this one keeps
  the naive O(w*r) per-column query, which is 70-100x slower
  than the Xor filters (see PR FastFilter#54 benchmark).
@amikumar91

Copy link
Copy Markdown
Contributor Author

@thomasmueller

Agreed on both points — implemented:

  • Renamed RibbonFilter RibbonNaiveFilter (and FilterType.RIBBON / TestFilterType.RIBBONRIBBON_NAIVE), so RibbonFilter/RIBBON stays free for a future ICML-backed implementation with fast queries.
  • Added class-level Javadoc citing Dietzfelbinger & Walzer (ESA 2019) and Dillinger & Walzer (arXiv:2103.02515), plus a note on why this is the "naive" query variant.

Please review again when you get a chance, thanks!

@amikumar91
amikumar91 marked this pull request as ready for review July 30, 2026 18:06
@thomasmueller
thomasmueller merged commit 334e394 into FastFilter:master Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ribbon Filter

2 participants