feat(fuse): support partitioned table layout - #20143
Conversation
github-actions
Bot
added
the
pr-feature
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a9fc8177e0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d19ad30b9c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3cc8d455b7
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. You're on a roll. Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
|
Findings P3 statement.rs (line 1177): table options before PARTITION BY are rejected only when ENGINE=FUSE is explicit. If the engine is omitted, the parser allows CREATE TABLE t(a INT) ROW_PER_BLOCK=1 PARTITION BY(a), and the binder later defaults that same table to Fuse (table.rs (line 588)). The same semantic table is rejected when written as ENGINE=FUSE ROW_PER_BLOCK=1 PARTITION BY(a). Either allow this order for Fuse consistently or reject it after resolving the default engine. |
There was a problem hiding this comment.
This is a great approach overall.
However, interleaved partition fragments cannot currently be consolidated by compaction. This can happen within a single parallel or distributed write, or accumulate across multiple writes.
For example, A, B, A, where both A segments belong to the same partition. Since compaction only groups adjacent segments from the same partition, these fragments may persist and increase segment and metadata overhead.
Maybe we need a way to consolidate non-adjacent fragments from the same partition while preserving the required segment order.
# Conflicts: # src/query/ast/src/parser/statement.rs # src/query/service/src/interpreters/interpreter_table_rename_column.rs # src/query/service/src/interpreters/interpreter_table_show_create.rs # src/query/sql/src/planner/binder/ddl/table.rs # src/query/storages/common/table_meta/src/table/table_keys.rs # src/query/storages/fuse/src/fuse_table.rs # src/query/storages/fuse/src/operations/append.rs # src/query/storages/fuse/src/operations/common/meta/mutation_log.rs # src/query/storages/fuse/src/operations/common/processors/transform_block_writer.rs # src/query/storages/fuse/src/operations/common/processors/transform_mutation_aggregator.rs # src/query/storages/fuse/src/operations/common/processors/transform_serialize_block.rs # src/query/storages/fuse/src/operations/recluster.rs # src/query/storages/fuse/src/pruning/fuse_pruner.rs
# Conflicts: # src/query/storages/fuse/src/operations/common/processors/transform_mutation_aggregator.rs
- sqllogictest: verify SET OPTIONS partition_by rejection, MODIFY COLUMN partition-key rejection, unrelated-filter no-pruning, DELETE+compact boundary preservation, and CTAS partition boundary on initial write - unit: partition_values/same_partition fallback cases (None stats, mismatched key id, min!=max prefix) - unit: PartitionPruner::should_keep returns true conservatively when segment has no cluster statistics
b9e77b6
4 participants
I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
Fuse tables currently have no table-level partition layout, so data rewrites cannot preserve partition boundaries and pruning cannot use exact partition values. This PR adds
PARTITION BYfor Fuse tables.Tests
Coverage includes parser and validation behavior, partition layout preservation, compaction and mutation behavior, exact expression pruning, strict and inclusive ranges, reversed comparisons, OR predicates, multiple partition keys, pruning counts, and query result correctness.
Type of change
This change is