Skip to content

[python] Use Native REST catalog branch operations - #10619

Merged
JingsongLi merged 4 commits into
apache:masterfrom
JingsongLi:codex/native-rest-branches
Oct 17, 2026
Merged

JingsongLi merged 4 commits into
apache:masterfrom
JingsongLi:codex/native-rest-branches

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Use Rust core for PyPaimon's existing REST branch operations, following Java's RESTCatalog and CatalogBranchManager semantics.

Depends on apache/paimon-rust#1141. Native CI continues to build Rust main; merge that PR first.

Changes

  • Delegate REST create/drop/fast-forward with write.native.enabled, and listing with read.native.enabled. Existing CatalogBranchManager calls use the same route.
  • Reuse resolved REST configuration without a second /config request. Preserve the live endpoint, owning authentication provider, headers and User-Agent. Translate structured Native errors through the existing public exception mappings.
  • Reuse the owning DLF provider's valid STS cache and refresh policy, passing the current credentials to Rust for each call. V4 request signing stays in Rust core. Cached credentials continue working when their source file or metadata service is temporarily unavailable.
  • Preserve custom Session transport behavior before delegation, including hooks, adapters, credentials, TLS configuration and non-default timeouts. Once Native execution begins, never replay an operation through Python.
  • Align strict branch creation with Java: list only for ignore_if_exists; preserve unknown 404 resource errors. Align REST rename with Java's unsupported operation and remove the invented REST rename endpoint/DTO.
  • Reject empty/dot URL components before dispatch. Document that creation without a tag creates an empty branch.
  • Add real HTTP and Native regressions for V4 MACs/actions/STS, encoded identifiers, headers, text responses, retries, credential refresh, custom transport behavior and uncertain publication. Shared Parquet/Blob data remains intact after branch operations.

Verification

  • Native REST branch and existing branch write/lifecycle suites: 109 passed, exercising 161 Native plans, 162 reads, 140 writes and 84 commits.
  • CatalogBranchManager, REST branch, branch DTO and DLF signer suites: 82 passed.
  • Rust dependency verification: 104 API unit tests, 188 REST integration tests and 83 binding/catalog/GIL/table tests passed.
  • Changed-file flake8 with dev/cfg.ini and git diff --check: passed. Rust core/binding Clippy and rustfmt also passed.

Tests used a locally built binding from Rust #1141. Repeated review repaired STS cache loss and transport/header differences. Paimon REST authenticates each request, so no Python/Rust cookie or session synchronization is added. The final independent pass found no confirmed P0/P1/P2 issues under this scope.

The mock REST service acknowledges fast-forward; request semantics are covered by Rust protocol tests and Java comparison, without claiming a production server's snapshot switching was tested here.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit 76cc485 into apache:master Oct 17, 2026
10 of 15 checks passed
sundapeng added a commit to sundapeng/paimon that referenced this pull request Oct 17, 2026
The REST surface had drifted three ways:

- RESTApi sends replaceTable (POST .../tables/{table}/replace),
  listSchemasPaged (GET .../tables/{table}/schemas) and loadSchema
  (GET .../tables/{table}/schemas/{version}), but the spec listed none of
  them. Add the routes plus ReplaceTableRequest, TableSchema,
  ListSchemasResponse and GetSchemaResponse.
- apache#8023 retired the renameBranch REST endpoint because the server-side
  rename is not atomic, and apache#10619 removed it from pypaimon, but the spec
  still listed it. Drop the route and RenameBranchRequest from the spec.
- pypaimon registered 55 of Java's 74 openapi-v4 actions. Give it the
  same table in the same order, since both resolve the first match, and
  check that against DLFOpenApiActions.java.
sundapeng added a commit to sundapeng/paimon that referenced this pull request Oct 17, 2026
The spec had drifted from RESTApi in two ways:

- RESTApi sends replaceTable (POST .../tables/{table}/replace),
  listSchemasPaged (GET .../tables/{table}/schemas) and loadSchema
  (GET .../tables/{table}/schemas/{version}), but the spec listed none of
  them. Add the routes plus ReplaceTableRequest, TableSchema,
  ListSchemasResponse and GetSchemaResponse.
- apache#8023 retired the renameBranch REST endpoint because the server-side
  rename is not atomic, and apache#10619 removed it from pypaimon, but the spec
  still listed it. Drop the route and RenameBranchRequest from the spec.
sundapeng added a commit to sundapeng/paimon that referenced this pull request Oct 17, 2026
The spec had drifted from RESTApi in two ways:

- RESTApi sends replaceTable (POST .../tables/{table}/replace),
  listSchemasPaged (GET .../tables/{table}/schemas) and loadSchema
  (GET .../tables/{table}/schemas/{version}), but the spec listed none of
  them. Add the routes plus ReplaceTableRequest, TableSchema,
  ListSchemasResponse and GetSchemaResponse.
- apache#8023 retired the renameBranch REST endpoint because the server-side
  rename is not atomic, and apache#10619 removed it from pypaimon, but the spec
  still listed it. Drop the route and RenameBranchRequest from the spec.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants