Skip to content

[python] Expand Native update input parity coverage - #10632

Merged
JingsongLi merged 3 commits into
apache:masterfrom
JingsongLi:codex/native-nested-update-inputs
Oct 17, 2026
Merged

JingsongLi merged 3 commits into
apache:masterfrom
JingsongLi:codex/native-nested-update-inputs

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Expand parity coverage for existing public PyPaimon update inputs that need companion Native core fixes: MAP entries, encoded Union values, fixed-size binary strings, Float16 strings, and exact/Java-compatible decimal conversion.

Depends on apache/paimon-rust#1149. Native CI continues to build Rust main. This PR stays draft until the core change is merged.

Brief change log

  • Cover batch/stream row-ID updates, incremental updates, predicate array/callable assignments, matched upserts and MERGE source-column assignments, only where Python already accepts the input.
  • Exercise MAP-to-ARRAY/ROW input adaptation at the root and inside ROW/ARRAY/MAP containers, renamed fields, slices, independent chunks, NULLs and empty containers.
  • Check fixed-size binary UTF-8/empty strings, invalid UTF-8 rejection, exact Float16 text, and Decimal128/256 floating rounding.
  • Cover Decimal32/64/128/256 exact rescaling and zero/NULL across extreme physical scales without relaxing overflow or fractional-loss checks.
  • Exercise dense/sparse Union primitive/ARRAY/ROW children, nonstandard IDs, inactive invalid values, dictionary NULL indices versus active NULLs, NULL parents, missing ROW fields, root MAP constructor scope, and unaligned foreign buffers.
  • Assert safe predicate/MERGE assignments still reject unsupported Union casts, append checks remain strict, and rejected updates do not commit.
  • Verify Python/Native reads, stable row IDs and untouched values. Fail on unintended Python update/read fallback when Native is selected.

Only tests are added. Python production code and public APIs remain unchanged; matching, normalization and conversion stay in Rust core, with private layout handling at its Arrow C Data boundary.

Review follow-up

  • Fix Decimal256 source precision 39..76 and coefficients wider than i128 when assigning FLOAT/DOUBLE. Use complete decimal coefficients while retaining Java compact/wide rules for valid logical DECIMAL inputs.
  • Normalize ordinary nested STRUCT siblings and active Union children along with sparse Union slices; avoid applying nested offsets twice.
  • Defer NULL/non-nullable-child validation until parent and child coordinates are both normalized, including masked STRUCT and FixedSizeList values.
  • Add 264 public-API regression cases. All three findings were reproduced before the fix; subsequent independent review found no remaining actionable issues. Java 8 BigDecimal results were checked directly.

Tests

  • Old Native failures reproduced while the corresponding Python API reference cases passed.
  • All 210 MAP/container reference cases and 795 scalar/encoded reference cases passed before Native verification.
  • Built the companion extension before Native tests; no build/install overlapped Native tests.
  • New and existing nested update, selected-value upsert, match-key, Blob, MERGE and row-shape regressions: 2669 passed, 2 skipped, with Native plan/read/write/commit enabled. Counters: 3243 plans, 4036 reads, 1451 writes, 147 commits.
  • Flake8 using dev/cfg.ini and diff checks passed.

@JingsongLi JingsongLi changed the title [python] Cover Native updates from Arrow MAP containers [python] Expand Native update input parity coverage Oct 17, 2026
@JingsongLi
JingsongLi marked this pull request as ready for review October 17, 2026 14:59

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit f3f1f95 into apache:master Oct 17, 2026
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants