Repository navigation
fix(checkpointing): set the unsafe_rbg PRNG impl in generate_param_only_checkpoint - #6026
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses a JAX PRNG implementation mismatch that caused checkpoint loading failures when using the Orbax v1 API. By ensuring the PRNG configuration matches the format used during checkpoint creation, the fix resolves shape compatibility errors and restores the finetune-to-decode pipeline. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
🤖 Hi @ryanliuCienet, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
|
🤖 I'm sorry @ryanliuCienet, but I was unable to process your request. Please see the logs for more details. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
065696d to
5f0418e
Compare
There was a problem hiding this comment.
Can we remove this unit test as generate_param_only_checkpoint_test.py also test the same line of code, so we are technically not losing coverage?
There was a problem hiding this comment.
Done — removed tests/unit/generate_param_only_checkpoint_prng_test.py.
5f0418e to
f84f7d0
Compare
f84f7d0 to
5b1fc4e
Compare
# Description
Root cause: #c41b2ae3a ("Refactor MaxText's checkpointing system to primarily
use the Orbax v1 API") replaced the v0 full-state restore in
`_load_linen_checkpoint_into_nnx` (`PyTreeRestore(item=linen_abstract,
partial_restore=True)`) with `ocp.load(root, linen_abstract)`. That exposed a
pre-existing mismatch in `generate_param_only_checkpoint`: it restores a full
training state with `load_full_state_path`, but unlike `train.py` (and
`decode.py`, `to_maxtext.py`, `forward_pass_logit_checker.py`, ...) it never
set `jax_default_prng_impl`, so it builds its restore target with JAX's default
`threefry2x32` keys while the checkpoint was written with `unsafe_rbg` keys.
The `rngs.*.key` arrays that #fbb35123a persists under `nnx_aux` therefore
disagree in shape -- `(num_layers, 2)` requested vs `(num_layers, 4)` stored.
The v0 restore accepted that and the conversion kept passing; the v1 load
rejects it and the conversion has failed ever since:
ValueError: Requested shape: (18, 2) is not compatible with the stored shape: (18, 4)
This breaks the finetune -> param-only -> decode flow in
`tests/end_to_end/tpu/{gemma/2b,llama2/7b,gemma/7b,llama2/70b}` (b/576849994)
at the first step that reads the fine-tuned checkpoint, so everything after it
in those scripts is unreachable.
Set the same PRNG impl as the other entry points before `pyconfig.initialize`.
The mismatch is on the caller's side -- the checkpoint and the v1 loader are
both right -- hence the fix here rather than in `checkpointing.py`.
The integration test did not catch this because it runs `train_main` and
`generate_param_only_ckpt_main` in one process, so the convert step inherited
`unsafe_rbg` from training. It now resets the impl to JAX's default before the
convert step, modelling the separate-process flow of the end-to-end scripts,
and adds a `cpu_only` entry point using a local checkpoint directory so CPU CI
exercises the flow and covers `generate_param_only_checkpoint.main`.
Fixes b/576849994
# Tests
- `pylint` 10.00/10, `pyink --check` clean on both files.
- `tests/integration/generate_param_only_checkpoint_test.py::test_param_ckpt_generation_with_dot_product_cpu`: passed.
- E2E on v5p-8 via the `maxtext_end_to_end` DAG with a test image built from
this branch: passed for gemma-2b and gemma-7b.
# Checklist
Before submitting this PR, please make sure (put X in square brackets):
- [X] I have performed a self-review of my code. For an optional AI review, add the `gemini-review` label.
- [X] I have necessary comments in my code, particularly in hard-to-understand areas.
- [X] I have run end-to-end tests and provided workload links above if applicable.
- [X] I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in [our documentation](https://maxtext.readthedocs.io/en/latest/development.html#adding-new-documentation-files).
5b1fc4e to
48c44ce
Compare
Description
c41b2ae3a ("Refactor MaxText's checkpointing system to primarily use the Orbax v1 API", #5083) replaced the v0 full-state restore in
_load_linen_checkpoint_into_nnx(PyTreeRestore(item=linen_abstract, partial_restore=True)) withocp.load(root, linen_abstract). That exposed a pre-existing mismatch ingenerate_param_only_checkpoint: it restores a full training state withload_full_state_path, but unliketrain.py(anddecode.py,to_maxtext.py,forward_pass_logit_checker.py, ...) it never setjax_default_prng_impl, so it builds its restore target with JAX's defaultthreefry2x32keys while the checkpoint was written withunsafe_rbgkeys. Therngs.*.keyarrays that fbb35123a ("[NNX] Checkpoint: persist and restore rngs/dropout across resumes", #4343) persists undernnx_auxtherefore disagree in shape,(num_layers, 2)requested vs(num_layers, 4)stored. The v0 restore accepted that and the conversion kept passing; the v1 load rejects it and the conversion has failed ever since:This breaks the finetune -> param-only -> decode flow in
tests/end_to_end/tpu/{gemma/2b,llama2/7b,gemma/7b,llama2/70b}at the first step that reads the fine-tuned checkpoint, so everything after it in those scripts is unreachable (including the logit-checker tokenizer failure fixed in #6025).Changes
src/maxtext/utils/generate_param_only_checkpoint.py: setjax_default_prng_impl=unsafe_rbgbeforepyconfig.initialize, the same as the other entry points. The mismatch is on the caller's side (the checkpoint and the v1 loader are both right), hence the fix here rather than incheckpointing.py.tests/integration/generate_param_only_checkpoint_test.py: the test did not catch this because it runstrain_mainandgenerate_param_only_ckpt_mainin one process, so the convert step inheritedunsafe_rbgfrom training. It now resets the impl to JAX's default before the convert step, modelling the separate-process flow of the end-to-end scripts.BUGS: b/576849994
Tests
Integration test flow on CPU
run_e2e_test_flowfrom the amendedtests/integration/generate_param_only_checkpoint_test.py(train 1 step ->generate_param_only_checkpoint-> decode) run on CPU in the stable imagemaxtext_jax_stable:2026-10-27withhardware=cpu,attention=dot_product,DECOUPLE_GCLOUD=TRUE:generate_param_only_checkpoint+ the amended test: fails at the convert step withValueError: Requested shape: (1, 2) is not compatible with the stored shape: (1, 4)(1-layer test model).The
tpu_onlytest itself runs in CI once the PR ispull ready.pylint10.00/10,pyink --checkclean on both files.E2E on TPU v5p-8
maxtext_end_to_endDAG (dev Composer env), clusterauto-v5p-8-bodaborg, imagemaxtext_jax_stable:2026-10-27+ this branch, #6024 and #6025 (maxtext_testing_liuryan:testing_all-6ef6677e1). The failing step is the secondgenerate_param_only_checkpointcall in each script (fine-tuned full state -> param-only,force_unroll=true).gemma/2b/test_gemma.sh(maxtext-stable-gemma-2b-v5p-8)Requested shape: (18, 2) ... (18, 4),EXIT_CODE=1EXIT_CODE=0llama2/7b/test_llama2_7b.sh(maxtext-stable-llama2-7b-v5p-8)Requested shape: (32, 2) ... (32, 4),EXIT_CODE=1llama_mistral_mixtral_orbax_to_hf, a separate pre-existing issue, fixed separately)gemma/7b/2_test_gemma.sh(chained,maxtext-stable-gemma-7b-v5p-8)EXIT_CODE=0Logs (Cloud Logging, project
cloud-tpu-multipod-dev):Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.