feat(query): refresh TopN statistics on append by KKould · Pull Request #20091 · databendlabs/databend

KKould · 2026-06-30T21:04:01Z

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Improve TopN freshness for append workloads. For non-transactional append commits, Databend now collects column TopN in the write path and refreshes table snapshot statistics when the previous snapshot statistics are still fresh.
This uses the existing table TopN options: configure analyze_top_n_columns and set analyze_top_n_size > 0 to enable collection for selected columns. If TopN options are disabled or cannot be safely refreshed, the optimizer falls back to the existing statistics behavior until a full ANALYZE.
Keep append TopN refresh out of explicit transactions for now. Transaction commits can collapse intermediate snapshot lineage, so transaction-aware TopN invalidation/regeneration will be handled separately instead of expanding this PR.
Reuse the existing statistics metadata flow instead of creating a separate TopN path: block/segment metadata carries column_top_n, commit meta aggregates it, and snapshot statistics are written together with the new snapshot.
Skip oversized string scalar values with LARGE_STRING_BYTES_THRESHOLD so one abnormal string value does not drop the whole TopN column.
Part of Track TopN freshness and runtime skew join follow-ups #20087.
Related to Handle HLL and TopN staleness after DELETE #20100.

Implementation

Let ColumnTopN carry its own capacity, which avoids passing top_n_size through every merge/add call.
Extend append block/segment/commit metadata to carry BlockTopN alongside HLL.
Merge append TopN into TableSnapshotStatistics only when the existing snapshot statistics match the current snapshot row count and predecessor snapshot id.
Limit append TopN refresh to autocommit append commits; explicit transactions skip refresh until table snapshot stats invalidation semantics are defined for transaction-local row-changing mutations.
Keep analyze and append collection on the same TopN builder path.

Tests

Unit Test
Logic Test
Benchmark Test
No Test - Explain why

Type of change

Bug Fix (non-breaking change which fixes an issue)
New Feature (non-breaking change which adds functionality)
Breaking Change (fix or feature that could cause existing functionality not to work as expected)
Documentation Update
Refactoring
Performance Improvement
Other (please describe):

This change is

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0e098ba9f4

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

KKould · 2026-07-02T01:24:52Z

@codex review

chatgpt-codex-connector · 2026-07-02T01:32:57Z

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: 6064b76024

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

KKould · 2026-07-02T06:31:21Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ef82699cc7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…eshness # Conflicts: # src/query/storages/fuse/src/fuse_table.rs # src/query/storages/fuse/src/io/write/block_statistics_writer.rs # src/query/storages/fuse/src/io/write/stream/block_builder.rs # src/query/storages/fuse/src/operations/analyze/analyze_ndv_meta.rs # src/query/storages/fuse/src/operations/analyze/analyze_state_sink.rs # src/query/storages/fuse/src/operations/analyze/collect_ndv_source.rs # src/query/storages/fuse/src/operations/commit.rs # src/query/storages/fuse/src/operations/common/processors/sink_commit.rs # src/query/storages/fuse/src/operations/common/processors/transform_mutation_aggregator.rs # tests/sqllogictests/suites/base/09_fuse_engine/09_0053_analyze_top_n.test

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 01acc8535c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

KKould · 2026-07-03T06:42:42Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 49905f7d45

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

KKould · 2026-07-08T02:51:28Z

@codex review

chatgpt-codex-connector · 2026-07-08T02:59:28Z

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 0a29de2fe0

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…eshness

…eshness # Conflicts: # src/query/storages/fuse/src/operations/commit.rs

zhang2014 · 2026-07-13T02:17:22Z

@zhyass please help review.

sundy-li · 2026-07-13T03:25:29Z

mutation will affect the statistics

zhyass

For Block HLL, when a new block is created, a block-level HLL is generated. These block HLLs are stored in a segment-level file with the prefix _hs. This way, when analyzing the table, we only need to read and aggregate these statistics files.
For TOPN, the table statistics file is updated whenever a write occurs. Which is somewhat different from the previous approach.

KKould · 2026-07-15T05:20:05Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 25b0715d00

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

KKould · 2026-07-15T08:05:07Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 34ff36e058

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…eshness # Conflicts: # src/query/storages/fuse/src/operations/commit.rs # src/query/storages/fuse/src/operations/common/processors/sink_commit.rs # src/query/storages/fuse/src/operations/common/processors/transform_mutation_aggregator.rs

github-actions Bot added the pr-feature this PR introduces a new feature to the codebase label Jun 30, 2026

KKould marked this pull request as draft June 30, 2026 21:08

chatgpt-codex-connector Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread src/query/storages/common/table_meta/src/meta/statistics.rs Outdated

feat(query): refresh topn statistics on append

6064b76

KKould force-pushed the feat/topn-append-freshness branch from 0e098ba to 6064b76 Compare July 2, 2026 01:14

KKould requested review from SkyFan2002, zhang2014 and zhyass July 2, 2026 01:25

KKould self-assigned this Jul 2, 2026

KKould marked this pull request as ready for review July 2, 2026 05:11

KKould marked this pull request as draft July 2, 2026 05:40

KKould force-pushed the feat/topn-append-freshness branch 3 times, most recently from a9d49cd to ef82699 Compare July 2, 2026 06:26

chatgpt-codex-connector Bot reviewed Jul 2, 2026

View reviewed changes

Comment thread src/query/storages/fuse/src/operations/commit.rs Outdated

refactor(query): simplify append topn stats commit

24e9fb2

KKould force-pushed the feat/topn-append-freshness branch from ef82699 to 24e9fb2 Compare July 2, 2026 07:16

KKould added 2 commits July 2, 2026 18:57

fix: add missing topn capacity in selectivity tests

01acc85

KKould marked this pull request as ready for review July 3, 2026 01:05

chatgpt-codex-connector Bot reviewed Jul 3, 2026

View reviewed changes

Comment thread src/query/storages/fuse/src/operations/commit.rs Outdated

KKould added 2 commits July 3, 2026 10:14

fix(storage): drop stale CMS on append TopN refresh

8739ef7

fix(storage): don't seed TopN from missing stats

49905f7

chatgpt-codex-connector Bot reviewed Jul 3, 2026

View reviewed changes

Comment thread src/query/storages/fuse/src/operations/commit.rs

fix: drop stale histograms on append topn refresh

3fbe302

KKould added 3 commits July 8, 2026 18:09

fix(storage): refresh topn for added nullable columns

0a29de2

Merge remote-tracking branch 'upstream/main' into feat/topn-append-fr…

cde3bb6

…eshness

Merge remote-tracking branch 'upstream/main' into feat/topn-append-fr…

9563e67

…eshness # Conflicts: # src/query/storages/fuse/src/operations/commit.rs

zhang2014 requested a review from forsaken628 July 13, 2026 02:36

zhyass reviewed Jul 13, 2026

View reviewed changes

Comment thread src/query/storages/fuse/src/operations/common/processors/sink_commit.rs Outdated

Comment thread src/query/storages/fuse/src/io/write/block_statistics_writer.rs

Comment thread src/query/storages/fuse/src/operations/commit.rs Outdated

KKould added 5 commits July 14, 2026 11:54

feat(fuse): cache TopN statistics per segment

c915731

revert: cache TopN statistics per segment

9d43af4

feat(fuse): cache per-block TopN in segment statistics

42bb4d8

fix(fuse): address statistics review feedback

7492e96

Merge branch 'main' into feat/topn-append-freshness

25b0715

chatgpt-codex-connector Bot reviewed Jul 15, 2026

View reviewed changes

Comment thread src/query/storages/fuse/src/operations/commit.rs Outdated

fix(fuse): refresh stats metadata for TopN-only appends

34ff36e

chatgpt-codex-connector Bot reviewed Jul 15, 2026

View reviewed changes

Comment thread src/query/storages/fuse/src/io/write/block_statistics_writer.rs

KKould requested a review from zhyass July 16, 2026 06:20

zhyass approved these changes Jul 19, 2026

View reviewed changes

KKould added this pull request to the merge queue Jul 20, 2026

Merged via the queue into databendlabs:main with commit d1681a7 Jul 20, 2026
91 checks passed

KKould deleted the feat/topn-append-freshness branch July 20, 2026 09:57

Uh oh!

Conversation

KKould commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Implementation

Tests

Type of change

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

KKould commented Jul 2, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 2, 2026

Uh oh!

KKould commented Jul 2, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

KKould commented Jul 3, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

KKould commented Jul 8, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 8, 2026

Uh oh!

zhang2014 commented Jul 13, 2026

Uh oh!

sundy-li commented Jul 13, 2026

Uh oh!

zhyass left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

KKould commented Jul 15, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

KKould commented Jul 15, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

KKould commented Jun 30, 2026 •

edited

Loading

zhyass left a comment •

edited

Loading