Skip to content

[python] Use native upserts for equivalent Arrow input layouts - #10624

Merged
JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/native-upsert-arrow-layouts
Oct 17, 2026
Merged

JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/native-upsert-arrow-layouts

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Purpose

Use Native execution for Arrow upsert inputs that PyPaimon already supports: large-string match keys, partition fields and nested values, plus equivalent fixed-size binary keys. The current exact type comparison sends these operations back to Python even though they represent the same logical table values.

Depends on apache/paimon-rust#1144. Native CI continues building apache/paimon-rust main, so the Rust change must merge first.

  • Reuse the existing Arrow schema compatibility policy for supplied upsert columns. Core owns normalization, partition-scoped matching, last-write-wins deduplication and file writing. Numeric conversions, different timestamp zones and other non-equivalent types keep their existing dispatch behavior.
  • Reuse read_arrow_schema when combining DE column bunches. File readers already return runtime-nullable nested types; reapplying the stricter logical schema could make an otherwise valid read fail.
  • Reuse the core-backed write normalizer at the Python row-ID writer boundary to restore declared nested write types and reject visible NULLs in required children. Keep NULL parent ROW values valid.
  • Close both the mapped batch iterator and its underlying merged reader on validation or writing errors, even while the exception traceback is retained.
  • Add Native/Python parity tests covering batch/stream upserts, multiple Arrow chunks, nested ROW/ARRAY/MAP strings, duplicate and NULL keys, partition isolation, partial writes, row identity, REST-backed writes, and required nested fields in row-ID and predicate updates.

The scope is read/write/update execution. There are no catalog metadata transport changes or new public APIs, options or dependencies.

Tests

Built the Rust bindings from apache/paimon-rust#1144 and ran with Native planning, reading, writing and committing enabled:

  • New layout cases plus chunked row-ID writer tests: 42 passed before the final reader-close regression was added.
  • Final regression selection (native_upsert_layout_test, table_update_by_row_id_chunked_test, table_update_test, data_evolution_test, native_blob_update_test): 383 passed, 24 subtests passed.
  • Broader upsert/nested update/BLOB/DE/concatenated reader selection: 568 passed, 2 skipped, 24 subtests passed.
  • Flake8 for every changed Python file and git diff --check.
  • The original Native fallback and core LargeUtf8 rejection were reproduced before the fixes. The new reader-close test failed in both validation-error and writer-error cases before explicitly closing the inner iterator.
  • Independent review of type eligibility, nested constraints and resource ownership; the identified reader-close issue was repaired and re-reviewed.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit 89a7ec3 into apache:master Oct 17, 2026
15 checks passed
sundapeng added a commit to sundapeng/paimon that referenced this pull request Oct 17, 2026
apache#10624 made the same change on master, so this commit now only conflicts
with it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants