Add Ribbon filter implementation - #54
Merged
thomasmueller merged 5 commits intoJul 30, 2026
Merged
Conversation
Implements the standard (non-homogeneous, w=64) Ribbon filter construction and membership query described in issue FastFilter#33, following Xor8's structure (hash-derive, build-with-retries, query-by-recombine).
Mirrors Xor8's ByteBuffer/OutputStream serialize + static deserialize convention. Unlike Xor8, numSlots/numStarts must be stored explicitly (not re-derived from size) since the construction retry loop can grow the slot count beyond the simple sizing formula.
Setting is ignored the same way XOR_8/XOR_16/XOR_PLUS_8 ignore it here: TestAllFilters.test() hardcodes setting=10 for every TestFilterType, which would exceed RibbonFilter's 8-bit (byte-storage) ceiling if passed through.
Contributor
|
It looks good to me! Given that this is a slow implementation (queries are very slow), what about using e.g. RibbonNaiveFilter (RIBBON_NAIVE) or RibbonSlowFilter or something similar? That way, the implementation can be retained later (one a faster implementation is available), which I think would make sense to better understand the algorithm. |
Contributor
Yes I think that would make sense! |
Reserves the RibbonFilter/RIBBON name for a future ICML-backed implementation with fast queries; this one keeps the naive O(w*r) per-column query, which is 70-100x slower than the Xor filters (see PR FastFilter#54 benchmark).
Contributor
Author
|
Agreed on both points — implemented:
Please review again when you get a chance, thanks! |
amikumar91
marked this pull request as ready for review
July 30, 2026 18:06
thomasmueller
approved these changes
Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add Ribbon filter implementation
What
Adds a standard (non-homogeneous) Ribbon filter,
org.fastfilter.ribbon.RibbonFilter, implementing theFilterinterface and registered asFilterType.RIBBON/TestFilterType.RIBBON.Closes #33
Why
A Ribbon filter is a retrieval data structure: for every key
xin the set, solving a banded linear system over GF(2) yields a small functionf(x)returning anr-bit fingerprint. It's structurally the closest existing analog in this repo toXor8/XorBinaryFuse8— hash a key, build with retries, query by re-deriving the hash and recombining — but the combine step is a dot product against a Gaussian-eliminated band rather than an XOR of three fingerprint slots.References:
Scope of this PR
Fixed ribbon width
w = 64(singlelong, no 128-bit/SIMD variant),firstCoeffAlwaysOne = true, row-majorbyte[] solutionstorage (one value per slot, mirroringXor8'sbyte[] fingerprints), configurableresultBits(default 8, capped at 8 by the byte-per-slot layout). Construction retries with a new seed on linear-dependence failure, and grows the slot-count overhead factor if all seed attempts at a size fail — the same retry shape asXor8's constructor loop, just with an extra outer loop for slot growth.Out of scope, called out here as separately-reviewable follow-ups:
w = 128/ SIMD-friendly widthsO(w · r)per-column loop; the benchmark below shows this is the main cost relative to the Xor filters, and ICML is the natural fix.numStarts = ceil(size * (1 + overheadFactor))formula, growingoverheadFactorby 1.5x on repeated failure)Testing
RibbonFilterTest: zero false negatives at sizes 10, 1,000, and 1,000,000; false-positive rate assertion (< 0.01, generous margin around the expected1/256 ≈ 0.0039) at 1,000,000 keys.RibbonFilterrow to the existing parameterizedxor/SerializationTest(round-trip via bothByteBufferand stream, buffer-too-small handling, multi-round serialization, etc.) rather than a new test file, since its constructor/deserializer signatures already fit that harness.mvn testpasses locally: 129 tests, 0 failures, 0 errors — including the full pre-existing suite (RegressionTests,TestAllFilters,SmallSetTest,StringFilters, etc.), confirming no regressions to other filter types.Quick local comparison (1,000,000 keys,
setting=10where applicable):Space is competitive with
XorBinaryFuse8at a similar target fpp, and construction time is in the same ballpark. Lookup is ~70-100x slower than the Xor filters — expected, since this PR intentionally uses the naive per-column dot product instead of ICML; that's the first candidate for a follow-up perf PR.Checklist
RibbonFilter.javaimplementsFilter: banding (on-the-fly Gaussian elimination), back-substitution, queryFilterType/TestFilterTypeupdated withRIBBONRibbonFilterTest: zero false negatives + fpp in range, multiple sizesByteBuffer+ stream)mvn testpasses locally (129/129)