Skip to content

fix(checkpointing): set the unsafe_rbg PRNG impl in generate_param_only_checkpoint - #6026

Merged
copybara-service[bot] merged 1 commit into
mainfrom
ryan/fix-param-only-ckpt-prng-impl
Nov 13, 2026
Merged

copybara-service[bot] merged 1 commit into
mainfrom
ryan/fix-param-only-ckpt-prng-impl

Conversation

@ryanliuCienet

@ryanliuCienet ryanliuCienet commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Description

c41b2ae3a ("Refactor MaxText's checkpointing system to primarily use the Orbax v1 API", #5083) replaced the v0 full-state restore in _load_linen_checkpoint_into_nnx (PyTreeRestore(item=linen_abstract, partial_restore=True)) with ocp.load(root, linen_abstract). That exposed a pre-existing mismatch in generate_param_only_checkpoint: it restores a full training state with load_full_state_path, but unlike train.py (and decode.py, to_maxtext.py, forward_pass_logit_checker.py, ...) it never set jax_default_prng_impl, so it builds its restore target with JAX's default threefry2x32 keys while the checkpoint was written with unsafe_rbg keys. The rngs.*.key arrays that fbb35123a ("[NNX] Checkpoint: persist and restore rngs/dropout across resumes", #4343) persists under nnx_aux therefore disagree in shape, (num_layers, 2) requested vs (num_layers, 4) stored. The v0 restore accepted that and the conversion kept passing; the v1 load rejects it and the conversion has failed ever since:

ValueError: Requested shape: (18, 2) is not compatible with the stored shape: (18, 4)

This breaks the finetune -> param-only -> decode flow in tests/end_to_end/tpu/{gemma/2b,llama2/7b,gemma/7b,llama2/70b} at the first step that reads the fine-tuned checkpoint, so everything after it in those scripts is unreachable (including the logit-checker tokenizer failure fixed in #6025).

Changes

  • src/maxtext/utils/generate_param_only_checkpoint.py: set jax_default_prng_impl=unsafe_rbg before pyconfig.initialize, the same as the other entry points. The mismatch is on the caller's side (the checkpoint and the v1 loader are both right), hence the fix here rather than in checkpointing.py.
  • tests/integration/generate_param_only_checkpoint_test.py: the test did not catch this because it runs train_main and generate_param_only_ckpt_main in one process, so the convert step inherited unsafe_rbg from training. It now resets the impl to JAX's default before the convert step, modelling the separate-process flow of the end-to-end scripts.

BUGS: b/576849994

Tests

Integration test flow on CPU

run_e2e_test_flow from the amended tests/integration/generate_param_only_checkpoint_test.py (train 1 step -> generate_param_only_checkpoint -> decode) run on CPU in the stable image maxtext_jax_stable:2026-10-27 with hardware=cpu, attention=dot_product, DECOUPLE_GCLOUD=TRUE:

  • Image's unfixed generate_param_only_checkpoint + the amended test: fails at the convert step with ValueError: Requested shape: (1, 2) is not compatible with the stored shape: (1, 4) (1-layer test model).
  • This branch: passes through decode.

The tpu_only test itself runs in CI once the PR is pull ready. pylint 10.00/10, pyink --check clean on both files.

E2E on TPU v5p-8

maxtext_end_to_end DAG (dev Composer env), cluster auto-v5p-8-bodaborg, image maxtext_jax_stable:2026-10-27 + this branch, #6024 and #6025 (maxtext_testing_liuryan:testing_all-6ef6677e1). The failing step is the second generate_param_only_checkpoint call in each script (fine-tuned full state -> param-only, force_unroll=true).

Script Before (current prod image) This PR
gemma/2b/test_gemma.sh (maxtext-stable-gemma-2b-v5p-8) Requested shape: (18, 2) ... (18, 4), EXIT_CODE=1 conversion passes; decode and the logit check run; script EXIT_CODE=0
llama2/7b/test_llama2_7b.sh (maxtext-stable-llama2-7b-v5p-8) Requested shape: (32, 2) ... (32, 4), EXIT_CODE=1 conversion passes; decode and the first logit check run (the script later fails in llama_mistral_mixtral_orbax_to_hf, a separate pre-existing issue, fixed separately)
gemma/7b/2_test_gemma.sh (chained, maxtext-stable-gemma-7b-v5p-8) not reached in prod: the chained tasks time out on the drained cluster (b/576852990) conversion passes; script EXIT_CODE=0

Logs (Cloud Logging, project cloud-tpu-multipod-dev):

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a JAX PRNG implementation mismatch that caused checkpoint loading failures when using the Orbax v1 API. By ensuring the PRNG configuration matches the format used during checkpoint creation, the fix resolves shape compatibility errors and restores the finetune-to-decode pipeline.

Highlights

  • PRNG Implementation Fix: Updated generate_param_only_checkpoint.py to set jax_default_prng_impl to unsafe_rbg before initialization, ensuring compatibility with checkpoints written using that implementation.
  • Integration Test Update: Modified the integration test to explicitly reset the PRNG implementation to threefry2x32 before the checkpoint generation step, accurately simulating the separate-process environment of production workflows.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions github-actions Bot added the skip-ci-tests Add this label to the PR to skip running CI tests label Oct 29, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @ryanliuCienet, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 I'm sorry @ryanliuCienet, but I was unable to process your request. Please see the logs for more details.

@codecov

codecov Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@ryanliuCienet ryanliuCienet changed the title Set unsafe_rbg PRNG impl in generate_param_only_checkpoint fix(checkpointing): set the unsafe_rbg PRNG impl in generate_param_only_checkpoint Oct 29, 2026
@ryanliuCienet
ryanliuCienet force-pushed the ryan/fix-param-only-ckpt-prng-impl branch from 065696d to 5f0418e Compare October 29, 2026 09:13

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove this unit test as generate_param_only_checkpoint_test.py also test the same line of code, so we are technically not losing coverage?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — removed tests/unit/generate_param_only_checkpoint_prng_test.py.

@ryanliuCienet
ryanliuCienet force-pushed the ryan/fix-param-only-ckpt-prng-impl branch from 5f0418e to f84f7d0 Compare November 12, 2026 05:05
@ryanliuCienet
ryanliuCienet force-pushed the ryan/fix-param-only-ckpt-prng-impl branch from f84f7d0 to 5b1fc4e Compare November 12, 2026 06:49
# Description

Root cause: #c41b2ae3a ("Refactor MaxText's checkpointing system to primarily
use the Orbax v1 API") replaced the v0 full-state restore in
`_load_linen_checkpoint_into_nnx` (`PyTreeRestore(item=linen_abstract,
partial_restore=True)`) with `ocp.load(root, linen_abstract)`. That exposed a
pre-existing mismatch in `generate_param_only_checkpoint`: it restores a full
training state with `load_full_state_path`, but unlike `train.py` (and
`decode.py`, `to_maxtext.py`, `forward_pass_logit_checker.py`, ...) it never
set `jax_default_prng_impl`, so it builds its restore target with JAX's default
`threefry2x32` keys while the checkpoint was written with `unsafe_rbg` keys.
The `rngs.*.key` arrays that #fbb35123a persists under `nnx_aux` therefore
disagree in shape -- `(num_layers, 2)` requested vs `(num_layers, 4)` stored.
The v0 restore accepted that and the conversion kept passing; the v1 load
rejects it and the conversion has failed ever since:

    ValueError: Requested shape: (18, 2) is not compatible with the stored shape: (18, 4)

This breaks the finetune -> param-only -> decode flow in
`tests/end_to_end/tpu/{gemma/2b,llama2/7b,gemma/7b,llama2/70b}` (b/576849994)
at the first step that reads the fine-tuned checkpoint, so everything after it
in those scripts is unreachable.

Set the same PRNG impl as the other entry points before `pyconfig.initialize`.
The mismatch is on the caller's side -- the checkpoint and the v1 loader are
both right -- hence the fix here rather than in `checkpointing.py`.

The integration test did not catch this because it runs `train_main` and
`generate_param_only_ckpt_main` in one process, so the convert step inherited
`unsafe_rbg` from training. It now resets the impl to JAX's default before the
convert step, modelling the separate-process flow of the end-to-end scripts,
and adds a `cpu_only` entry point using a local checkpoint directory so CPU CI
exercises the flow and covers `generate_param_only_checkpoint.main`.

Fixes b/576849994

# Tests

- `pylint` 10.00/10, `pyink --check` clean on both files.
- `tests/integration/generate_param_only_checkpoint_test.py::test_param_ckpt_generation_with_dot_product_cpu`: passed.
- E2E on v5p-8 via the `maxtext_end_to_end` DAG with a test image built from
  this branch: passed for gemma-2b and gemma-7b.

# Checklist

Before submitting this PR, please make sure (put X in square brackets):
- [X] I have performed a self-review of my code. For an optional AI review, add the `gemini-review` label.
- [X] I have necessary comments in my code, particularly in hard-to-understand areas.
- [X] I have run end-to-end tests and provided workload links above if applicable.
- [X] I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in [our documentation](https://maxtext.readthedocs.io/en/latest/development.html#adding-new-documentation-files).
@ryanliuCienet
ryanliuCienet force-pushed the ryan/fix-param-only-ckpt-prng-impl branch from 5b1fc4e to 48c44ce Compare November 12, 2026 07:18
@NuojCheng NuojCheng removed the skip-ci-tests Add this label to the PR to skip running CI tests label Nov 13, 2026
@copybara-service
copybara-service Bot merged commit 48c44ce into main Nov 13, 2026
111 of 112 checks passed
@copybara-service
copybara-service Bot deleted the ryan/fix-param-only-ckpt-prng-impl branch November 13, 2026 05:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants