Skip to content

[core] Support REST catalog version management - #1142

Merged
JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/native-rest-version-management
Oct 17, 2026
Merged

JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/native-rest-version-management

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Expose REST tag management, paginated tag listing, latest-snapshot metadata and guarded rollback through the Rust Catalog API and Python bindings, matching Java semantics. These capabilities already exist in PyPaimon; PyPaimon continues to use its Python REST client.

Changes

  • Add a core Instant type and typed rollback request matching Java's snapshot/tag wire protocol, omitting absent fromSnapshot and tableId guards.
  • Expose snapshot loading, rollback and tag pagination through Catalog and implement them in RESTCatalog without resolving targets or publishing changes through local storage.
  • Preserve Java's query and pagination semantics, including empty page tokens, opaque prefixes, missing tags versus empty pages, and distinct table/tag/snapshot errors.
  • Preserve REST errors, including HTTP 501 and the missing latest snapshot of an empty table. Match resource types with Java's case sensitivity.
  • Accept successful mutation responses without requiring a JSON body. Never replay failed mutations.
  • Preserve base/delta manifest-list sizes in snapshot metadata so the Native bridge does not discard fields Python already uses.
  • Add thin Python catalog bindings and typed metadata wrappers; update DataFusion REST rollback callers to forward the loaded table UUID.

The scope is existing PyPaimon functionality. No new dependencies are introduced.

Verification

  • cargo +1.98.0 test --quiet --locked -p paimon --test rest_branch_test --test rest_catalog_test — 153 tests.
  • Native binding HTTP tests in bindings/python/tests/test_catalog_versions.py — 36 tests.
  • TZ=UTC cargo +1.98.0 test --quiet --locked -p paimon-datafusion --test procedures test_rollback — 4 tests. The timestamp procedure test interprets a local TIMESTAMP value as an epoch; UTC matches the CI environment.
  • cargo +1.98.0 clippy --locked -p paimon -p paimon-datafusion -p pypaimon_rust --all-targets --features fulltext,vortex -- -D warnings.
  • Rustfmt and Python binding test Flake8 checks.
  • Exercised the new bindings from PyPaimon's real HTTP tests with Native planning, reading, writing and committing enabled, including V4 signatures, table replacement/head rejection, error classification, and no replay or cleanup after publication failures.

CI follow-up

The unit job exposed a stale tag error-mapping test that still expected lowercase resource types. Update it to Java's exact uppercase resource names and add assertions for unknown/lowercase resource types, missing latest snapshots and HTTP 501 preservation. The public interfaces are unchanged.

  • Full paimon library tests with fulltext,vortex: 4276 passed, 6 ignored.
  • Core library/test Clippy with -D warnings and rustfmt passed.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit 53dfa2c into apache:main Oct 17, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants