Skip to content

Restore jax_use_shardy_partitioner in conftest.py and configure it in train_compile.py. - #6403

Open
copybara-service[bot] wants to merge 1 commit into
mainfrom
test_1023876587
Open

copybara-service[bot] wants to merge 1 commit into
mainfrom
test_1023876587

Conversation

@copybara-service

@copybara-service copybara-service Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Restore jax_use_shardy_partitioner in conftest.py and configure it in train_compile.py.

Description

In cpu-integration under pytest-xdist (e.g. TPU Docker Images Pipeline Run 44139847036, job 134250571837), tests/unit/train_compile_test.py::TrainCompile::test_moe_deepseek_unscanned_bf16 crashes the worker with Fatal Python error: Aborted inside lowered.compile() when scheduled on the same xdist worker immediately after tests/integration/train_tests.py::TrainTests::test_base_model_shardy_false:

  1. PR Move slow glm5_3 and deepseek unscanned compile tests from cpu-unit to cpu-integration #6177 moved TrainCompile::test_moe_deepseek_unscanned_bf16 from cpu-unit into cpu-integration, placing it in the same pytest-xdist pool as TrainTests::test_base_model_shardy_false.
  2. test_base_model_shardy_false invokes train.main(..., "shardy=False"), which sets jax.config.update("jax_use_shardy_partitioner", False) globally in the worker process (train.py:1632).
  3. Unlike jax_default_prng_impl, jax_remove_size_one_mesh_axis_from_type, and LIBTPU_INIT_ARGS, jax_use_shardy_partitioner was not restored by tests/conftest.py (added in PR Restore JAX PRNG and LIBTPU_INIT_ARGS via pytest fixtures in conftest.py #6227), and train_compile.py did not call jax.config.update("jax_use_shardy_partitioner", config.shardy).

This change:

  • Saves and restores jax_use_shardy_partitioner in _restore_jax_prng_and_libtpu_env() in tests/conftest.py.
  • Calls jax.config.update("jax_use_shardy_partitioner", config.shardy) in train_compile.py (get_topology_mesh and main), matching train.py and decode.py.
  • Adds regression tests in tests/unit/marker_propagation_test.py (test_restore_jax_use_shardy_partitioner) and tests/unit/train_compile_test.py (test_get_topology_mesh_updates_shardy_partitioner).

Tests

  • pytest tests/unit/marker_propagation_test.py -k test_restore_jax_use_shardy_partitioner
  • pytest tests/unit/train_compile_test.py -k test_get_topology_mesh_updates_shardy_partitioner
  • blaze test //third_party/py/maxtext/tests:train_compile_google_test

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@codecov

codecov Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/maxtext/trainers/pre_train/train_compile.py 0.00% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

PiperOrigin-RevId: 1023876587

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant