Skip to content

[python] Extend Native upsert keys and preserve decimal index predicates - #10635

Merged
JingsongLi merged 3 commits into
apache:masterfrom
JingsongLi:codex/native-read-write-parity
Oct 18, 2026
Merged

JingsongLi merged 3 commits into
apache:masterfrom
JingsongLi:codex/native-read-write-parity

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Enable Native for additional existing PyPaimon upsert key layouts and make DECIMAL predicate results consistent across Python and Native.

Changes

  • Allow the Native upsert bridge to pass equivalent numeric and Binary/LargeBinary key families to core. This includes integer/floating/decimal cross-family keys, UInt64 boundaries and Decimal32/64/128/256 encodings. Core validates exact identity; selected payloads and unmatched-row append validation retain the original arrays.
  • Fix Python bitmap global-index filtering for finer-scale DECIMAL literals. For example, DECIMAL(10,2) != Decimal('1.235') previously rounded the lookup literal to 1.24 and discarded that matching row. Decline index evaluation when the serializer cannot represent the literal exactly, leaving the original predicate for data filtering, consistent with Native.
  • Add regressions covering dictionary and composite/partition keys, duplicate-key last-write-wins across chunks, NULLs, batch/stream writes, rejected lossy conversions, strict unmatched appends, and a two-byte LargeBinary key after a 2 GiB mmap prefix.
  • Cover valid LargeBinary arrays whose NULL slot retains more than 2 GiB of hidden storage, for both all-NULL and mixed NULL/non-NULL keys. Native matching must ignore those bytes and preserve row IDs and untouched columns.
  • Add real BTree/bitmap global-index tests for compact/wide decimals, comparisons, membership/NULL/range/compound predicates, independent Native plan/read settings, and predicate updates/callables/deletes.
  • Test full-row numeric upserts separately from match-only keys, including 2^53+2 matching successfully while its original integer-to-double payload still violates the assignment safe range. Test mixed matched/unmatched failures leave all rows unchanged, and PK deduplicate/partial-update reads apply finer-scale decimal predicates to the merged row values.

Dependency

Depends on apache/paimon-rust#1150. Keep this draft until it is merged. Native CI continues to build Rust main; no extension pin or fallback is added.

Verification

2101 tests passed against the companion extension with Native plan/read/write execution counters checked. This includes the new decimal/upsert regressions and existing encoded-input, Dictionary MERGE, MERGE INTO, writer-capability, decimal-expression, and index-evaluator suites. Flake8 passed for all changed Python files. Independent review rounds completed, including the bounded exact numeric algorithm, NULL storage, and payload-identity mutation checks.

Only current read/write/update paths are changed; no metadata features or legacy Java compatibility is added.

@JingsongLi
JingsongLi marked this pull request as ready for review October 18, 2026 00:46

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit e711c1c into apache:master Oct 18, 2026
14 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants