Skip to content

feat(query): prune blocks for internal column subqueries - #20140

Closed
SkyFan2002 wants to merge 1 commit into
databendlabs:mainfrom
SkyFan2002:internal_runtime_filter
Closed

feat(query): prune blocks for internal column subqueries#20140
SkyFan2002 wants to merge 1 commit into
databendlabs:mainfrom
SkyFan2002:internal_runtime_filter

Conversation

@SkyFan2002

@SkyFan2002 SkyFan2002 commented Jul 13, 2026

Copy link
Copy Markdown
Member

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

  • Optimize positive _block_name IN (subquery) predicates for Fuse tables into dynamic block pruning.
  • Execute the subquery build side first, publish its exact block-location set, and prune the probe scan before reading blocks.
  • Remove the redundant row-level hash join while preserving IN semantics for duplicate, empty, and NULL build results.
  • Keep the optimization restricted to safe left-semi joins over the Fuse _block_name internal column and broadcast the build side in distributed execution.

Implementation

  • Annotate eligible MarkJoin-to-LeftSemi rewrites with DynamicBlockPrune and preserve the annotation through optimizer rules.
  • Build a dedicated physical plan and sink that collects exact string locations from the subquery.
  • Integrate the published locations into both row-oriented and column-oriented Fuse block-pruning paths.
  • Wait for dynamic locations with query-abort handling before enumerating matching blocks.

Validation

  • cargo check -p databend-common-sql
  • cargo test -p databend-common-catalog dynamic_block_prune_filter_publishes_exact_locations
  • cargo test -p databend-query --test it test_dynamic_block_prune_from_in_subquery
  • cargo clippy -p databend-common-sql -p databend-common-storages-fuse -p databend-query --lib -- -D warnings
  • cargo fmt --all -- --check
  • git diff HEAD^ --check

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

The regression coverage includes one selected block, duplicate build rows, an empty build result, NULL build values, and pipeline assertions that the dynamic-prune sink replaces the hash join.

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

Rollout risk

No migration, configuration toggle, or backfill is required. Queries that do not match the guarded Fuse _block_name pattern retain the existing join path.


This change is Reviewable

@github-actions github-actions Bot added the pr-feature this PR introduces a new feature to the codebase label Jul 13, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b986cf3c97

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/query/storages/fuse/src/operations/read_partitions.rs
Comment thread src/query/service/src/physical_plans/physical_join.rs Outdated
@SkyFan2002
SkyFan2002 force-pushed the internal_runtime_filter branch 2 times, most recently from 41179d2 to 3904e49 Compare July 13, 2026 06:27
@SkyFan2002

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3904e49a9a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/query/service/src/physical_plans/physical_join.rs
@SkyFan2002
SkyFan2002 force-pushed the internal_runtime_filter branch from 3904e49 to b9c90c8 Compare July 13, 2026 06:55
@SkyFan2002

Copy link
Copy Markdown
Member Author

@codex review

@SkyFan2002
SkyFan2002 force-pushed the internal_runtime_filter branch from b9c90c8 to 0623e24 Compare July 13, 2026 07:03

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b9c90c8df4

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/query/service/src/physical_plans/physical_join.rs Outdated
@SkyFan2002

Copy link
Copy Markdown
Member Author

@codex review

@SkyFan2002
SkyFan2002 force-pushed the internal_runtime_filter branch from 0623e24 to 3014824 Compare July 13, 2026 07:06
@SkyFan2002

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3014824be8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/query/catalog/src/table_context/runtime_filter.rs Outdated
@SkyFan2002
SkyFan2002 force-pushed the internal_runtime_filter branch from 3014824 to afe3e89 Compare July 13, 2026 07:21
@SkyFan2002

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: afe3e89fa7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/query/service/src/physical_plans/physical_join.rs
@SkyFan2002
SkyFan2002 force-pushed the internal_runtime_filter branch from afe3e89 to 60957a7 Compare July 13, 2026 07:33
@SkyFan2002

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 60957a75f6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/query/service/src/physical_plans/physical_join.rs Outdated
@SkyFan2002
SkyFan2002 force-pushed the internal_runtime_filter branch 2 times, most recently from 586691f to 368fd70 Compare July 13, 2026 07:45
@SkyFan2002

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 368fd7022a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/query/service/src/physical_plans/physical_join.rs Outdated
@SkyFan2002
SkyFan2002 force-pushed the internal_runtime_filter branch from 368fd70 to 394e1bd Compare July 13, 2026 07:54
@SkyFan2002

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 394e1bd2a8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/query/service/src/physical_plans/physical_join.rs Outdated
@SkyFan2002
SkyFan2002 force-pushed the internal_runtime_filter branch from 394e1bd to d48b308 Compare July 13, 2026 08:07
@SkyFan2002

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d48b3084f6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/query/service/src/physical_plans/physical_join.rs Outdated
@SkyFan2002
SkyFan2002 force-pushed the internal_runtime_filter branch from d48b308 to da50f18 Compare July 13, 2026 08:19
@SkyFan2002

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: da50f1816e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@SkyFan2002
SkyFan2002 force-pushed the internal_runtime_filter branch from da50f18 to aae8038 Compare July 13, 2026 09:10
@SkyFan2002

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: aae803884d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@SkyFan2002
SkyFan2002 force-pushed the internal_runtime_filter branch from aae8038 to a3a4b1a Compare July 13, 2026 10:06
@SkyFan2002

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: a3a4b1a7ba

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@SkyFan2002
SkyFan2002 force-pushed the internal_runtime_filter branch from a3a4b1a to 88a370d Compare July 13, 2026 11:12
@SkyFan2002

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 88a370d574

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@SkyFan2002
SkyFan2002 force-pushed the internal_runtime_filter branch from 88a370d to 97cd656 Compare July 13, 2026 12:03
@SkyFan2002

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: 97cd656f53

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@SkyFan2002
SkyFan2002 requested a review from zhang2014 July 13, 2026 12:14
@SkyFan2002 SkyFan2002 closed this Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-feature this PR introduces a new feature to the codebase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant