Repository navigation
[python] Use native upserts for equivalent Arrow input layouts - #10624
Merged
JingsongLi merged 2 commits intoOct 17, 2026
Merged
Conversation
sundapeng
added a commit
to sundapeng/paimon
that referenced
this pull request
Oct 17, 2026
apache#10624 made the same change on master, so this commit now only conflicts with it.
This was referenced Oct 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Use Native execution for Arrow upsert inputs that PyPaimon already supports: large-string match keys, partition fields and nested values, plus equivalent fixed-size binary keys. The current exact type comparison sends these operations back to Python even though they represent the same logical table values.
Depends on apache/paimon-rust#1144. Native CI continues building apache/paimon-rust
main, so the Rust change must merge first.read_arrow_schemawhen combining DE column bunches. File readers already return runtime-nullable nested types; reapplying the stricter logical schema could make an otherwise valid read fail.The scope is read/write/update execution. There are no catalog metadata transport changes or new public APIs, options or dependencies.
Tests
Built the Rust bindings from apache/paimon-rust#1144 and ran with Native planning, reading, writing and committing enabled:
native_upsert_layout_test,table_update_by_row_id_chunked_test,table_update_test,data_evolution_test,native_blob_update_test): 383 passed, 24 subtests passed.git diff --check.