Skip to content

[core] Fix VARIANT timestamp projection and scalar conversions - #1151

Merged
JingsongLi merged 1 commit into
apache:mainfrom
XiaoHongbo-Hope:codex/fix-variant-conversions
Oct 18, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
XiaoHongbo-Hope:codex/fix-variant-conversions

Conversation

@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor

Purpose

Fix incorrect VARIANT timestamp projections, missing Boolean-to-numeric casts, and decimal shredding overflow.

Brief change log

  • Convert timestamp microseconds to the requested Arrow unit; nanosecond overflow errors in strict mode and returns NULL in tolerant mode.
  • Support Boolean-to-integer, DOUBLE and DECIMAL extraction, consistent with Java.
  • Keep integers in residual storage when they cannot fit the shredded DECIMAL column.

Tests

Parquet round-trip regressions cover plain/shredded VARIANT, timestamp units and overflow, Boolean casts, and decimal residuals.

  • cargo test -p paimon --lib variant: 93 passed.
  • cargo test -p paimon --lib arrow::shredding: 49 passed (overlapping coverage).
  • cargo fmt --all -- --check and git diff --check passed.

API and Format

No public API or storage format changes.

Documentation

Not required for these correctness fixes.

@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review October 17, 2026 16:46

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit 5fa2c33 into apache:main Oct 18, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants