sparse-ngrams: report each query gram's index-folded bytes

[aneubeck]

Problem

QueryGrams emits only the compact NGram key. Callers that want to show a gram — tracing, snapshot tests, debugging output — cannot recover the text from it: grams longer than 3 bytes are hashed into the 24-bit payload, so Debug degrades to NGram('0x20baba', len=4).

Blackbird hit exactly this after switching its query-time extraction over to this crate: its iterator-tree snapshots went from

$[OR2[AND[*0=c:boring :20, c:art is g:40, 1=c:ing mi:20, ...

to

$[OR2[AND[*c:NGram('t w', len=3):40, 0=c:NGram('0x20baba', len=4):2, ...

Change

The state already holds the bytes: extract_gram builds the key from the packed content window. Align that window once and hand the consumer a &[u8] view of it. Consumers become FnMut(NGram, u32, Option<u8>, &[u8]):

arg	meaning
gram	the compact NGram key
end	position of the character just after the gram
follow	that following byte, when already fed
bytes	the gram's index-folded bytes, in reading order

bytes borrows a stack array that is live only for the duration of the call, so nothing is allocated per gram and the hot path is unchanged (the alignment shift was already being done inside from_window_masked).

NGram::from_window_masked is removed: aligning in extract_gram makes its masking redundant, since the left shift already drops the high bytes.

Compatibility

Breaking for the four QueryGrams emitters (append_char, append_byte, flush, consume_first). The indexing entrypoints (collect_sparse_grams*) are untouched — their callers already hold the content slice and can slice it themselves. Version bumped to 0.4.0.

Testing

emitted_follow_is_the_actual_next_char is extended (and renamed to emitted_follow_and_bytes_match_the_input) to assert, for every emitted gram, that the reported slice equals the corresponding input slice and that NGram::from_bytes(bytes) reproduces the emitted key. cargo test -p sparse-ngrams, cargo clippy --all-targets and cargo fmt --check are clean.

[Copilot]

Pull request overview

Adds index-folded source bytes to QueryGrams callbacks for readable diagnostics without per-gram allocation.

Changes:

• Extends callbacks with &[u8] and validates emitted bytes.
• Simplifies aligned-window NGram construction.
• Updates documentation and bumps version to 0.4.0.

Show a summary per file

File	Description
crates/sparse-ngrams/src/query.rs	Emits gram bytes and updates tests.
crates/sparse-ngrams/src/ngram.rs	Removes redundant masked-window helper.
crates/sparse-ngrams/README.md	Documents the callback API.
crates/sparse-ngrams/Cargo.toml	Bumps the breaking-release version.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 4/4 changed files
• Comments generated: 0
• Review effort level: Medium

[aneubeck]

Follow-up pushed: the reported bytes were not free for consumers that ignore them, and now they are.

What was wrong

Slicing the stack buffer with a runtime len leaves a bounds-check panic path in the generated code. That path is an observable side effect, so LLVM can't prove the buffer is dead even when the inlined consumer never touches it — the byte-swap and stack traffic survive.

Measured on aarch64 (--release, an append_byte loop whose consumer only sums gram.as_u32()), instruction counts for the whole function:

variant	instructions
0.3.0, no bytes reported	229
0.4.0, consumer ignores bytes	243
0.4.0, consumer reads bytes	247

So every consumer paid a 14-instruction tax, wanted or not. The asm confirmed the cause — a live core::slice::index::slice_index_fail call.

Fix

Clamp the slice length to MAX_SPARSE_GRAM_SIZE so the bound is statically provable. The panic path disappears and the buffer becomes dead code when unused:

variant	instructions
0.3.0, no bytes reported	229
0.4.0 + clamp, consumer ignores bytes	229
0.4.0 + clamp, consumer reads bytes	240

Ignoring consumers are now byte-for-byte identical to the pre-change codegen, and consumers that do read the bytes got cheaper too (247 → 240), since the removed check was on the hot path.

The clamp never changes the slice — len is always in 2..=MAX_SPARSE_GRAM_SIZE, which from_window already debug-asserts. Comment in the code explains why it's there so nobody "simplifies" it away.