Skip to content

[core] Preserve decimal predicates and equivalent Arrow match keys - #1150

Merged
JingsongLi merged 4 commits into
apache:mainfrom
JingsongLi:codex/native-read-write-parity
Oct 18, 2026
Merged

JingsongLi merged 4 commits into
apache:mainfrom
JingsongLi:codex/native-read-write-parity

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Move two existing PyPaimon read/update cases onto Native while preserving exact comparison and key identity.

Changes

  • Preserve DECIMAL predicate literals at their own precision/scale when they cannot be represented exactly at column scale. The core already compares DECIMAL values across scales, as Java's Decimal.compareTo does. Stored partition values continue to use strict column conversion.
  • Accept exact numeric keys across integer, floating, and DECIMAL families, as well as equivalent Binary/LargeBinary/FixedSizeBinary layouts. Keep original source arrays for selected payloads and unmatched-row append validation.
  • Compare decimal/floating keys by the exact binary floating value using bounded i256 arithmetic. This distinguishes Decimal('0.1') from DOUBLE 0.1, while accepting Decimal.from_float(0.1), negative decimal scales, and Decimal32/64/128/256 encodings. Integer/floating key matching permits exactly representable values such as 2^53+2 without relaxing payload assignment limits.
  • Rebase LargeBinary offsets to the visible byte range before narrowing, including small slices after a 2 GiB backing-buffer prefix. Share the existing STRING offset logic without broadening ordinary append schemas.
  • Ignore storage held by NULL LargeBinary key slots when it exceeds the 32-bit offset range. Compact valid bytes only in this case and reject genuinely oversized valid payloads before allocating the output.
  • Add binding/core regression coverage for finer/wider decimal literals, dictionary NULLs, floating rounding/overflow/underflow rejection, binary bytes, and sparse mmap slices.

Verification

  • Rust key normalization tests: 11 passed.
  • Write normalization tests, including STRING/Binary sparse mmap, hidden NULL storage, and visible-byte overflow rejection: 24 passed.
  • Python binding predicate tests: 78 passed.
  • Rust 1.98.0 Clippy for core and bindings (--lib --tests -- -D warnings): passed.
  • Companion PyPaimon integration/regression suite against this extension: 2101 passed, with Native plan/read/write execution counters checked. Covers batch/stream upserts, full-row and mixed matched/unmatched payload validation, UInt64 boundaries, predicate updates/deletes, BTree/bitmap global indexes, compact/wide decimals, and mixed plan/read modes. PK deduplicate and partial-update tests verify finer-scale decimal filtering observes merged rows across files.

No new public API, dependencies, or legacy Java compatibility paths.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit 619843d into apache:main Oct 18, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants