Skip to content

[python] Plan natively with pypaimon-rust releases that lack Plan.stats() - #10620

Closed
sundapeng wants to merge 2 commits into
apache:masterfrom
sundapeng:python-native-plan-stats-compat
Closed

sundapeng wants to merge 2 commits into
apache:masterfrom
sundapeng:python-native-plan-stats-compat

Conversation

@sundapeng

@sundapeng sundapeng commented Oct 17, 2026 •

Copy link
Copy Markdown
Member

Purpose

#10609 reads planning counters with rust_plan.stats(). pypaimon-rust 0.4.0, the latest release and the one the Python / Tests jobs install, has no Plan.stats(), so every native plan raises AttributeError and falls back to the Python scanner:

Native plan failed, falling back to the Python scanner: 'pypaimon_rust.datafusion.Plan' object has no attribute 'stats'

This made ContiguousWindowPlanTest.test_native_plan_retains_independent_payload_column_files fail on Python 3.10 to 3.13. #10624 moved that test to the Rust main job, but native planning with 0.4.0 still falls back. Native CI builds pypaimon-rust from main, which has stats(), so it does not see this.

This PR reads the counters only when the binding provides stats(), as _from_native_plan already does for snapshot_id. With an older binding the plan stays native and reports no native statistics, as before #10609.

It also adds Native CI / pypaimon-rust 0.4.0, which installs the released binding and runs the regular suite with native planning forced. The version is a matrix value, so later releases can be added next to it.

Tests

  • native_plan_test.py adds test_bindings_without_plan_stats_still_plan_natively, which fails without the fix.
  • With pypaimon-rust 0.4.0 installed and -m 'not native_plan', contiguous_window_plan_test.py fails before the fix and passes with it.
  • Native CI / pypaimon-rust 0.4.0 passes and plans natively 2402 times; before the fix the same suite plans natively twice.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a CI for 0.4.0

@sundapeng

Copy link
Copy Markdown
Member Author

Thanks, added in 4df684f. Native CI / pypaimon-rust 0.4.0 installs the released 0.4.0 from PyPI and runs the regular suite with native planning forced; tests marked native_plan need Rust main and stay in Native CI. It passes here with 2403 native plans. Run locally on master before the fix, the same suite fails test_native_plan_retains_independent_payload_column_files and plans natively only twice, since every other plan falls back on the missing Plan.stats(). The version is a matrix value, so the next release can be added next to it.

The Python / Native CI failure here is unrelated: apache/paimon-rust#1141 made Rust retry GET requests on 429 and 503, and four snapshot tests still expect a single 503 call. It fails the same way on master; #10626 fixes the tests.

@sundapeng sundapeng changed the title [python] Plan natively with pypaimon-rust releases that lack Plan.stats() [python] Fix the Python Tests and Native CI failures on master Oct 17, 2026
@sundapeng sundapeng changed the title [python] Fix the Python Tests and Native CI failures on master [python] Plan natively with pypaimon-rust releases that lack Plan.stats() Oct 17, 2026
@sundapeng sundapeng closed this Oct 17, 2026
@sundapeng sundapeng reopened this Oct 17, 2026
…ts()

apache#10609 calls rust_plan.stats(), which pypaimon-rust 0.4.0, the release
the Python test jobs install, does not have. The AttributeError sends
every native plan back to the Python scanner, so
ContiguousWindowPlanTest.test_native_plan_retains_independent_payload_column_files
fails on master. Read the counters only when the binding has them, as
snapshot_id already does; such plans report no native statistics.
Native CI builds pypaimon-rust from paimon-rust main, and Python Tests
skip native planning tests, so no job forces native planning through a
released binding. That is how apache#10609 broke 0.4.0 without notice: every
native plan failed on Plan.stats() and silently fell back to the Python
scanner.

Native CI / pypaimon-rust 0.4.0 installs the release from PyPI and runs
the regular suite with native planning forced. Tests marked native_plan
need Rust main, so they stay in the main job. On the previous commit
this job fails test_native_plan_retains_independent_payload_column_files
and plans natively only twice.
@sundapeng
sundapeng force-pushed the python-native-plan-stats-compat branch from 80a925e to 5fc3201 Compare October 17, 2026 12:00
@sundapeng

Copy link
Copy Markdown
Member Author

#10624 fixed the failing Python / Tests by moving test_native_plan_retains_independent_payload_column_files to the Rust main job, so CI no longer needs this. What remains is that opt-in native planning with pypaimon-rust 0.3.0 or 0.4.0 logs a warning and falls back to the Python scanner, with correct results, until a release with Plan.stats() becomes the minimum. Closing; it can be reopened if 0.4.0 compatibility is wanted.

@sundapeng sundapeng closed this Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants