Skip to content

[python] Expand Native upserts, Dictionary MERGE and Parquet vector updates - #10629

Merged
JingsongLi merged 4 commits into
apache:masterfrom
JingsongLi:codex/native-upsert-selected-values
Oct 17, 2026
Merged

JingsongLi merged 4 commits into
apache:masterfrom
JingsongLi:codex/native-upsert-selected-values

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Expand Native read/write/update execution for capabilities PyPaimon already supports, keeping matching and conversion in Rust core.

  • Keep Arrow upserts on Native when surviving matched payloads need existing row-ID conversions, or when ignored/discarded values do not satisfy the append schema. Newly appended rows retain ordinary write validation.
  • Qualify logical upsert keys across exact integer widths, decimal scales and temporal units, Dictionary encodings, and NULL-typed input. Rust validates values, overflow and precision; original input keys remain subject to their action's write rules.
  • Enable Dictionary MERGE keys, including Pandas categorical sources and partition keys. Rust decodes keys and applies SQL NULL/cardinality semantics; Python adds no matching implementation.
  • Enable row-ID, incremental, predicate, upsert, MERGE and shard updates for Parquet vector sidecars, including coexistence with BLOB and Paimon table MERGE sources. Reuse existing core writers and dedicated-file metadata.
  • Preserve split order when the Python shard updater uses Native reading. Parallel completion order otherwise associates update values with other row IDs. A regression reads actual files in reversed group completion order to reproduce the problem deterministically.

Matched values reuse the existing core row-ID conversions, including Binary/LargeBinary; ordinary append validation keeps its existing rules. No new dependency, public API, metadata path, or storage format.

Depends on apache/paimon-rust#1147. Native CI continues to build Rust main, so this PR remains a draft until the core changes are merged.

Tests

Built the Native extension from the companion Rust branch before starting Python tests, then ran the selected-value, key normalization, Dictionary MERGE, Parquet vector, existing upsert layout/row-shape, Blob row/update, nested update, general Native update/MERGE and shard update suites with PYPAIMON_TEST_NATIVE_PLAN=1, PYPAIMON_TEST_NATIVE_READ=1, PYPAIMON_TEST_NATIVE_WRITE=1 and PYPAIMON_TEST_NATIVE_COMMIT=1.

Result: 1322 passed, 2 skipped. Native counters: 1887 plans, 2236 reads, 809 writes, 257 commits. Flake8 with the repository config and git diff --check passed.

New tests compare Python/Native batch and stream execution, fail explicitly on unintended Python update fallbacks, and verify values, unchanged columns, partitions, stable row IDs and dedicated vector/BLOB metadata. Key coverage includes exact conversions, overflow/precision rejection, duplicate last-write-wins, target fan-out, NULL indices and NULL dictionary values, partition dictionaries, Dictionary MERGE cardinality and unconditional DELETE, plus original append type validation. 32 positive Python/Native batch and stream cases verify Binary/LargeBinary conversions for arbitrary bytes, NULLs, nested MAP/ROW values and constructor-shaped inputs, with stable row IDs.

Follow-up review fixes

  • Remove the upsert-specific Binary/LargeBinary restriction and its recursive policy plumbing in the companion core PR. The Python bridge adds no conversion policy; matched payloads reuse the established row-ID conversions.
  • Preserve Blob map transport and Java key serialization independently of value conversion: Dictionary keys and Dictionary payloads, exact integer widths, DECIMAL HALF_UP and TIME millisecond normalization. Normalization-induced duplicate keys still fail, and append types remain strict.
  • Handle exact decimal zero/NULL keys across extreme scale differences without Arrow power-table errors or i8 scale arithmetic overflow. Nonzero overflow and precision loss remain errors.
  • Add regression cases for those failures, NULL containers, mixed Blob map transport and unchanged row identities. Several independent review passes found no remaining confirmed issue after repairs.

@JingsongLi JingsongLi changed the title [python] Keep convertible Arrow upserts on Native [python] Expand Native upserts, Dictionary MERGE and Parquet vector updates Oct 17, 2026

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi marked this pull request as ready for review October 17, 2026 13:19
@JingsongLi
JingsongLi merged commit 847a717 into apache:master Oct 17, 2026
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants