Repository navigation
Use the configured PRNG impl in standalone_checkpointer - #6292
Conversation
|
🤖 Hi @NuojCheng, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses a compatibility issue where standalone_checkpointer.py failed to restore checkpoints due to a mismatch in PRNG implementation settings. By aligning the PRNG configuration with that used in training, the checkpointer can now correctly interpret the stored RNG keys, preventing shape mismatch errors. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
There was a problem hiding this comment.
📋 Review Summary
Verdict: No significant issues found.
Inferred spec & validation
- Sources: PR description; code before/after.
- Problem:
standalone_checkpointer.pyfails to restore NNX checkpoints fromtrain.pydue to mismatched PRNG implementations causing RNG key shape errors ((2, 2)vs(2, 4)).
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): standalone_checkpointer.py must set jax_default_prng_impl to config.prng_impl |
✅ | standalone_checkpointer_nnx_test.py::MainTest.test_main_uses_configured_prng_impl ✅ |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
train.py sets jax_default_prng_impl to config.prng_impl (unsafe_rbg by default), and NNX checkpoints store RNG keys whose shape depends on it. standalone_checkpointer never set it, so restoring a train.py checkpoint failed with "Requested shape: (layers, 2) is not compatible with the stored shape: (layers, 4)".
e6b14a8 to
d84d4eb
Compare
Stacked on #6026. Its base is #6026's branch, so the diff shows only this change. Once #6026 merges, retarget this PR to
main.Description
standalone_checkpointer.pycan't restore a checkpoint written bytrain.py:train.pysetsjax_default_prng_impltoconfig.prng_impl(unsafe_rbgby default), and NNX checkpoints store RNG keys whose shape depends on the impl:unsafe_rbgkeys areuint32[4], while JAX's defaultthreefry2x32keys areuint32[2].standalone_checkpointer.main()never set the impl, so it built its restore target with threefry keys and Orbax rejected the stored rbg keys.This change sets
jax_default_prng_implfromconfig.prng_implright after the config loads, the same valuetrain.pyends up with.#6026 fixes the same mismatch in
generate_param_only_checkpoint.py. This PR coversstandalone_checkpointer.py, which #6026 doesn't touch. One difference: it usesconfig.prng_implinstead of hard-codingunsafe_rbg, so it also matches checkpoints written with a non-defaultprng_impl.Tests
MainTest.test_main_uses_configured_prng_implchecks thatmain()appliesconfig.prng_implbeforecheckpoint_loopruns. It fails without the fix ('threefry2x32' != 'rbg') and passes with it. The rest oftests/unit/standalone_checkpointer_nnx_test.pypasses too (8 passed).standalone_checkpointeron that checkpoint directory in a new process. Before: fails with the error above. After: restores step 1, then saves and restores step 2.JAX_DEFAULT_PRNG_IMPLworkaround: train with checkpointing, convert withgenerate_param_only_checkpoint, then restore withstandalone_checkpointer.pre-commit run(codespell, pylint, pyink) passes on the changed files.Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.skip-ci-testsso only CPU/lint/docs checks run during review; a bot removes it after 2 approvals on a 1-commit PR to run TPU/GPU suites, or remove it yourself / ask a maintainer on fork PRs to run them sooner. On TPU7X, only the TPU tests that the PR adds or modifies run; add therun-tpu7xlabel to run the full TPU7X suite.)