Skip to content

Use the configured PRNG impl in standalone_checkpointer - #6292

Merged
copybara-service[bot] merged 1 commit into
mainfrom
chengnuojin-standalone-checkpointer-prng-impl
Nov 13, 2026
Merged

copybara-service[bot] merged 1 commit into
mainfrom
chengnuojin-standalone-checkpointer-prng-impl

Conversation

@NuojCheng

@NuojCheng NuojCheng commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #6026. Its base is #6026's branch, so the diff shows only this change. Once #6026 merges, retarget this PR to main.

Description

standalone_checkpointer.py can't restore a checkpoint written by train.py:

ValueError: Requested shape: (2, 2) is not compatible with the stored shape: (2, 4).

train.py sets jax_default_prng_impl to config.prng_impl (unsafe_rbg by default), and NNX checkpoints store RNG keys whose shape depends on the impl: unsafe_rbg keys are uint32[4], while JAX's default threefry2x32 keys are uint32[2]. standalone_checkpointer.main() never set the impl, so it built its restore target with threefry keys and Orbax rejected the stored rbg keys.

This change sets jax_default_prng_impl from config.prng_impl right after the config loads, the same value train.py ends up with.

#6026 fixes the same mismatch in generate_param_only_checkpoint.py. This PR covers standalone_checkpointer.py, which #6026 doesn't touch. One difference: it uses config.prng_impl instead of hard-coding unsafe_rbg, so it also matches checkpoints written with a non-default prng_impl.

Tests

  • The new MainTest.test_main_uses_configured_prng_impl checks that main() applies config.prng_impl before checkpoint_loop runs. It fails without the fix ('threefry2x32' != 'rbg') and passes with it. The rest of tests/unit/standalone_checkpointer_nnx_test.py passes too (8 passed).
  • End to end on CPU with a 2-layer llama2-style model: train 2 steps with checkpointing, then run standalone_checkpointer on that checkpoint directory in a new process. Before: fails with the error above. After: restores step 1, then saves and restores step 2.
  • With this PR stacked on fix(checkpointing): set the unsafe_rbg PRNG impl in generate_param_only_checkpoint #6026, the whole flow runs on CPU with no JAX_DEFAULT_PRNG_IMPL workaround: train with checkpointing, convert with generate_param_only_checkpoint, then restore with standalone_checkpointer.
  • pre-commit run (codespell, pylint, pyink) passes on the changed files.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable. (Note: New PRs get skip-ci-tests so only CPU/lint/docs checks run during review; a bot removes it after 2 approvals on a 1-commit PR to run TPU/GPU suites, or remove it yourself / ask a maintainer on fork PRs to run them sooner. On TPU7X, only the TPU tests that the PR adds or modifies run; add the run-tpu7x label to run the full TPU7X suite.)
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @NuojCheng, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a compatibility issue where standalone_checkpointer.py failed to restore checkpoints due to a mismatch in PRNG implementation settings. By aligning the PRNG configuration with that used in training, the checkpointer can now correctly interpret the stored RNG keys, preventing shape mismatch errors.

Highlights

  • PRNG Implementation Alignment: Updated standalone_checkpointer.py to set jax_default_prng_impl from the configuration, ensuring compatibility with checkpoints generated by train.py.
  • Regression Testing: Added a new unit test in tests/unit/standalone_checkpointer_nnx_test.py to verify that the main function correctly applies the configured PRNG implementation.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: No significant issues found.

Inferred spec & validation
  • Sources: PR description; code before/after.
  • Problem: standalone_checkpointer.py fails to restore NNX checkpoints from train.py due to mismatched PRNG implementations causing RNG key shape errors ((2, 2) vs (2, 4)).
Spec item Implementation Validating test
R1 (stated): standalone_checkpointer.py must set jax_default_prng_impl to config.prng_impl ✅ standalone_checkpointer_nnx_test.py::MainTest.test_main_uses_configured_prng_impl ✅

@codecov

codecov Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@NuojCheng NuojCheng removed the skip-ci-tests Add this label to the PR to skip running CI tests label Nov 13, 2026
train.py sets jax_default_prng_impl to config.prng_impl (unsafe_rbg by
default), and NNX checkpoints store RNG keys whose shape depends on it.
standalone_checkpointer never set it, so restoring a train.py checkpoint
failed with "Requested shape: (layers, 2) is not compatible with the
stored shape: (layers, 4)".
@NuojCheng
NuojCheng force-pushed the chengnuojin-standalone-checkpointer-prng-impl branch from e6b14a8 to d84d4eb Compare November 13, 2026 01:59
@NuojCheng
NuojCheng changed the base branch from main to ryan/fix-param-only-ckpt-prng-impl November 13, 2026 01:59
Base automatically changed from ryan/fix-param-only-ckpt-prng-impl to main November 13, 2026 05:04
@copybara-service
copybara-service Bot merged commit 76746c6 into main Nov 13, 2026
73 checks passed
@copybara-service
copybara-service Bot deleted the chengnuojin-standalone-checkpointer-prng-impl branch November 13, 2026 05:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants