refactor(storage): scale clustering information diagnostics

[zhyass]

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

This PR makes CLUSTERING_INFORMATION diagnostics scale to multi-million-block tables while preserving exact closed-interval overlap and depth semantics.

• Encode typed endpoints once and sort endpoint IDs with an MSD radix sorter.
• Compute Linear overlap/depth aggregates directly instead of materializing one result per block.
• Replace the Hilbert event-grid implementation with dense ranking, adaptive outer-axis partitioning, and exact partition-local sweeps.
• Run CPU-heavy sorting and sweeps in one bounded request-local Rayon pool (up to 8 workers), avoiding retained worker pools between requests.
• Keep CLUSTERING_STATISTICS focused on per-block depth and remove overlap work that only CLUSTERING_INFORMATION needs.
• Stream segment metadata in bounded chunks and retain only endpoint columns required by diagnostics.

Performance comparison

Release-mode synthetic A/B benchmarks used identical generated endpoint data and validated identical checksums/results. Times are medians; lower is better. The temporary benchmark instrumentation was removed from this PR.

Linear diagnostics: root/main vs this PR

Key shape	Blocks	Before	After	Speedup	Reduction
Date + String	10,000	4.188 ms	2.456 ms	1.71x	41.4%
UInt16 + UInt64	10,000	3.071 ms	0.953 ms	3.22x	69.0%
Date + String	100,000	80.512 ms	16.034 ms	5.02x	80.1%
UInt16 + UInt64	100,000	61.901 ms	6.836 ms	9.06x	89.0%
Date + String	1,000,000	1,466.476 ms	212.271 ms	6.91x	85.5%
UInt16 + UInt64	1,000,000	1,182.212 ms	164.697 ms	7.18x	86.1%

Hilbert adaptive partitioning: previous implementation vs this PR

5,383,590 rectangles, 8 workers, 3 rounds:

Distribution	Before	After	Speedup	Reduction
Random rectangles	46,084.172 ms	10,963.530 ms	4.20x	76.2%
High overlap	12,532.151 ms	4,191.668 ms	2.99x	66.6%
Low-cardinality X	31,792.279 ms	4,325.596 ms	7.35x	86.4%
Low-cardinality Y	23,010.803 ms	4,393.464 ms	5.24x	80.9%

A separate production-scale Linear sorting A/B over 5,383,590 blocks (10,767,180 endpoints, 8 workers, 7 rounds) reduced total diagnostics time from 2,631.641 ms to 2,066.595 ms (1.27x); endpoint sorting improved from 1,487.677 ms to 931.937 ms (1.60x), while encoding and sweep time remained effectively unchanged.

Implementation notes

• Endpoint IDs remain tied to the caller-provided alternating [min, max] layout; unstable sorting is safe because ties are grouped by encoded value.
• Starts are processed before ends at equal coordinates, preserving closed-interval boundary overlap.
• Hilbert partitions replicate only rectangles crossing partition boundaries and fall back to a single global sweep when the replica budget would be exceeded.
• Exact integer aggregates are retained until the response boundary, where averages are rounded to four decimal places.
• Empty snapshots and vector-only cluster keys preserve existing serialized response behavior.

Tests

• Unit Test
• Logic Test
• Benchmark Test
• No Test - Explain why

Validation performed:

• cargo test -p databend-common-storages-fuse --lib (129 passed)
• cargo clippy -p databend-common-storages-fuse --lib --tests -- -D warnings
• cargo test -p databend-common-storages-fuse statistics::endpoint_sort::tests --release --lib
• cargo test -p databend-common-storages-fuse statistics::hilbert::tests --release --lib
• cargo test -p databend-common-storages-fuse table_functions::clustering_information::tests --release --lib
• cargo check -p databend-common-storages-fuse --lib
• git diff --check

Focused coverage includes independent randomized differential checks for Linear and Hilbert diagnostics, closed-boundary semantics, malformed ranges, radix ordering, request-local worker limits, and the adaptive large-input Hilbert path.

Type of change

• Bug Fix (non-breaking change which fixes an issue)
• New Feature (non-breaking change which adds functionality)
• Breaking Change (fix or feature that could cause existing functionality not to work as expected)
• Documentation Update
• Refactoring
• Performance Improvement
• Other (please describe):

AI assistance

• AI usage: An AI coding agent helped analyze the clustering diagnostics, draft the implementation and tests, run validation and benchmarks, simplify the final diff, and draft this PR description.
• Responsible human: @zhyass
• The responsible human has read every line of this diff and can explain each change

---

This change is 

[github-actions]

Docker Image for PR

• tag: pr-20411-5621e0e-1788103435

note: this image tag is only available for internal use.