feat(storage): optimize ngram bloom index building

[KKould]

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

• Stream lowercase ngram generation without allocating a lowercase String, character-offset Vec, or per-row digest Vec
• Build ngram Bloom filters directly instead of retaining all distinct digests in a block-level HashSet
• Fold Bloom filter bitmaps to a power-of-two size selected from the observed occupancy and target false-positive rate
• Add the false_positive_rate NGRAM index option while keeping the default at 0.01
• Preserve compatibility with existing non-power-of-two Bloom filters

Implementation

• Directly borrows byte windows from ASCII rows without uppercase bytes and reuses a lowercase byte buffer for other ASCII rows
• Streams Unicode lowercase mappings through a sliding ngram window
• Changes filter builders to consume themselves during build() and makes add_digests() default to repeated add_digest() calls
• Estimates the distinct digest count from bitmap occupancy, then folds corresponding bitmap halves without introducing false negatives
• Uses a bit mask for probe positions on power-of-two filters and retains modulo as the legacy fallback
• Parses and validates finite false_positive_rate values in the open interval (0, 1)

The configured bloom_size remains the maximum filter size. The target false-positive rate controls probe count and the adaptive folding target. Workloads can explicitly relax it to trade pruning precision for smaller indexes; the benchmark below uses 0.8 for that specific long-query workload, while the general default remains 0.01.

Benchmark

The benchmark follows #17852 and uses the six Amazon Reviews Parquet files from 2010 through 2015:

• 39.2 GB input
• 135,589,433 rows
• Approximately 17 GB in review_body
• gram_size = 10
• bloom_size = 2097152
• Current branch: false_positive_rate = 0.8
• max_threads = 16

NGRAM INDEX idx1 (review_body)
    gram_size = 10
    bloom_size = 2097152
    false_positive_rate = 0.8

COPY

The without-NGRAM baseline uses the same current release binary, source files, 15-column schema, and thread count, but does not declare an NGRAM index. Fuse still writes its normal Bloom indexes in this baseline.

Year	main	current (0.8)	without NGRAM
2010	43.468 s	22.727 s	3.811 s
2011	47.821 s	31.647 s	4.725 s
2012	62.010 s	38.721 s	7.647 s
2013	115.030 s	64.249 s	12.551 s
2014	137.308 s	77.086 s	22.092 s
2015	103.279 s	60.229 s	23.622 s
Total	508.916 s	294.659 s	74.448 s

The ASCII fast path reduced COPY time by another approximately 12.3% compared with the previous implementation result of 336.015 seconds. The total COPY time decreased by approximately 42.1% compared with main.

The current NGRAM build takes approximately 3.96x the without-NGRAM baseline, compared with 6.84x on main. Measured above the 74.448-second baseline, this PR reduces the incremental NGRAM time from 434.468 seconds to 220.211 seconds, approximately 49.3%. The three runs produced 612, 611, and 610 blocks respectively, so small block-layout effects remain in the comparison.

Index and query

The query is the LIKE aggregation from #17852:

WHERE review_body LIKE '%The first track with Chris Botti is beautiful%'

Metric	main	current (0.8)
Rows	135,589,433	135,589,433
Blocks	612	611
Ngram index size	1,283,482,116 B	1,226,858,971 B
Folded filters	0	52 to 1 MiB; 559 remain 2 MiB
Candidate blocks after pruning	5	4

The generated block layouts differ, so the candidate-block counts and index sizes are not treated as direct query-performance comparisons. The current result confirms that folding is applied while the LIKE query continues to prune the large majority of blocks.

Tests

• Unit Test
• Logic Test
• Benchmark Test
• No Test - Pair with the reviewer to explain why

Type of change

• Bug Fix (non-breaking change which fixes an issue)
• New Feature (non-breaking change which adds functionality)
• Breaking Change (fix or feature that would cause existing functionality not to work as expected)
• Documentation Update
• Refactoring
• Performance Improvement
• Other (please describe):

AI assistance

• AI usage: An AI coding agent assisted with the ngram iterator, Bloom builder and folding implementation, tests, benchmark execution, and PR summary; the responsible human reviewed the final diff.
• Responsible human: @KKould
• The responsible human has read every line of this diff and can explain each change

---

This change is 

[github-actions]

🤖 CI Job Analysis (Retry 1)

Workflow: 33253569420

📊 Summary

• Total Jobs: 91
• Failed Jobs: 2
• Retryable: 0
• Code Issues: 2

❌ NO RETRY NEEDED

All failures appear to be code/test issues requiring manual fixes.

🔍 Job Details

• ❌ linux / sqllogic / standalone (standalone, 2c, hybrid): Not retryable (Code/Test)
• ❌ linux / sqllogic / standalone (standalone, 2c, http): Not retryable (Code/Test)

---

🤖 About

Automated analysis using job annotations to distinguish infrastructure issues (auto-retried) from code/test issues (manual fixes needed).

[sundy-li]

阻塞问题（P1）：本 PR 当前提交 26d2584 的 CI 失败不是基础设施问题。linux / sqllogic / standalone (standalone, 2c, http) 和 hybrid 都在 tests/sqllogictests/suites/mode/standalone/explain/index/explain_ngram_index.test:98 失败：EXPLAIN ... content LIKE '%月无声%' 预期读取 2 行、扫描 1 个 partition，实际读取 4 行、扫描 2 个 partitions。该变化与 src/query/storages/common/index/src/filters/bloom_filter.rs 中固定 probe 数和 adaptive folding 直接相关。请修复默认 false_positive_rate=0.01 下的 pruning 行为，或确认这是有意的 false-positive tradeoff 并同步更新该回归用例及相关测试；在 required checks 通过前不能合并。