Repository navigation
fix(ckpt-conversion): port llama_mistral_mixtral_orbax_to_hf to the NNX training state - #6035
ryanliuCienet wants to merge 1 commit into
Conversation
# Description
Root cause: #b1ac62ab0 ("NNX: flip pure_nnx/enable_nnx/pure_nnx_decoder
defaults to True") made `_read_train_checkpoint` -- which this standalone
script borrows from `generate_param_only_checkpoint` -- return the NNX state by
default: a flat `nnx.State` with the weights under `model` and the optimizer
under `optimizer`, instead of a Linen `TrainState`. #bbb843f1d ("[NNX] Delete
Linen (5/5)") then removed the Linen path altogether. The script was never
ported and still indexes the old layout, `training_state.params["params"]`, so
every conversion dies on the first weight it touches:
AttributeError: No attribute 'params' in State
In `tests/end_to_end/tpu/llama2/7b/test_llama2_7b.sh` this is the MaxText -> HF
step after the first logit check; it stayed hidden behind the earlier failures
in that script (the logit checker tokenizer, #<TOKENIZER_PR_NUMBER>, and the
`generate_param_only_checkpoint` PRNG mismatch, #<PRNG_PR_NUMBER>) and is only
reachable once those are in. `tests/end_to_end/tpu/llama3.1/8b/{2,3}_test_llama3.1_8b.sh`
run the same script.
Reshape the loaded state once with `train_state_nnx.to_checkpoint_dict` -- the
helper the checkpoint writer uses, so the result is exactly the on-disk Linen
layout -- and hand `convert_state_to_hf` the `["params"]["params"]` subtree it
was written against. The porting code is unchanged apart from the shorter
indexing; `convert_gpt3_ckpt_from_paxml` already loads its state the same way.
# Tests
- `pylint` 10.00/10, `pyink --check` clean.
- CPU (`JAX_PLATFORMS=cpu`, stable image): tiny llama2 (dense) and mixtral
(MoE, 2 experts) models built through `load_model_params` -- a fresh init is
the same `nnx.State` the restore path returns -- then `convert_state_to_hf`
with matching `MODEL_PARAMS_DICT` entries: every HF tensor name and shape as
expected, values equal to the MaxText arrays. The pre-fix script fails on the
same models with the `AttributeError` above.
- E2E on v5p-8 via the `maxtext_end_to_end` DAG with a test image built from
this branch: pending (fill in workload link for llama2-7b).
# Checklist
Before submitting this PR, please make sure (put X in square brackets):
- [X] I have performed a self-review of my code. For an optional AI review, add the `gemini-review` label.
- [X] I have necessary comments in my code, particularly in hard-to-understand areas.
- [X] I have run end-to-end tests and provided workload links above if applicable.
- [X] I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in [our documentation](https://maxtext.readthedocs.io/en/latest/development.html#adding-new-documentation-files).
|
🤖 Hi @ryanliuCienet, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request updates the MaxText to HuggingFace checkpoint conversion script to support the transition from Linen-based training states to the new NNX-based state. By reshaping the loaded NNX state into the legacy Linen layout, the conversion logic is restored to functionality without requiring significant changes to the downstream parameter mapping. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
There was a problem hiding this comment.
📋 Review Summary
Verdict: No significant issues found.
Inferred spec & validation
- Sources: PR description, code before/after.
- Problem:
_read_train_checkpointnow returns an NNXStateobject rather than a LinenTrainState, causingllama_mistral_mixtral_orbax_to_hfto crash withAttributeErrorwhen looking up theparamsattribute.
| Spec item | Implementation | Validating test |
|---|---|---|
| R1 (stated): Conversion script extracts weights successfully from the NNX state. | ✅ | test_llama2_7b.sh ✅ |
| R2 (stated): Extracted weights map to the HuggingFace layout correctly without dimension errors. | ✅ | test_llama2_7b.sh ✅ |
| C1 (inferred): Existing conversion logic for parameters remains unchanged. | ✅ preserved | test_llama2_7b.sh ✅ |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Description
b1ac62ab0 ("NNX: flip pure_nnx/enable_nnx/pure_nnx_decoder defaults to True", #3526) made
_read_train_checkpoint-- which this standalone script borrows fromgenerate_param_only_checkpoint-- return the NNX state by default: a flatnnx.Statewith the weights undermodeland the optimizer underoptimizer, instead of a LinenTrainState. bbb843f1d ("[NNX] Delete Linen (5/5): remove the pure_nnx/enable_nnx/pure_nnx_decoder config flags", #4361) then removed the Linen path altogether.llama_mistral_mixtral_orbax_to_hfwas never ported and still indexes the old layout,training_state.params["params"][...], so every conversion dies on the first weight it touches:In
tests/end_to_end/tpu/llama2/7b/test_llama2_7b.shthis is the MaxText -> HF step after the first logit check. It has never been visible in prod: the script fails earlier, at the logit checker's tokenizer load (b/579251787, #6025) and, before that, at thegenerate_param_only_checkpointPRNG mismatch (b/576849994, #6026). It is the next failure once those two are in.tests/end_to_end/tpu/llama3.1/8b/{2,3}_test_llama3.1_8b.shrun the same script.Changes
src/maxtext/checkpoint_conversion/standalone_scripts/llama_mistral_mixtral_orbax_to_hf.pyonly:load_model_state->load_model_params: reshape the loaded state once withtrain_state_nnx.to_checkpoint_dict-- the helper the checkpoint writer uses, so the result is exactly the on-disk Linen layout -- and return its["params"]["params"]subtree, i.e. what used to beTrainState.params["params"].convert_gpt3_ckpt_from_paxmlalready loads its state the same way.convert_state_to_hf(params, ...): the 16training_state.params["params"][...]lookups becomeparams[...]; the porting logic itself is unchanged.BUGS: b/579252268
Related: b/576849994 and b/579251787 (the PRNG and tokenizer failures that kept this step unreachable; fixed in #6026 and #6025).
Tests
CPU check
Stable image
maxtext_jax_stable:2026-10-27,JAX_PLATFORMS=cpu,pylint10.00/10 andpyink --checkclean.Tiny llama2 (dense) and mixtral (MoE, 2 experts) models built through
load_model_params-- a fresh init is the samennx.Statethe restore path returns -- thenconvert_state_to_hfwith matchingMODEL_PARAMS_DICTentries: every HF tensor name and shape as expected (21 dense / 29 MoE tensors), values equal to the MaxText arrays (embed_tokens,v_proj,down_proj, MoEw1spot-checked). The pre-fix script fails on the same models with theAttributeErrorabove.E2E on TPU v5p-8
maxtext_end_to_endDAG (dev Composer env), clusterauto-v5p-8-bodaborg, imagemaxtext_jax_stable:2026-10-27+ this branch, #6024, #6025 and #6026 (maxtext_testing_liuryan:testing_all-b22ff6314), taskmaxtext-stable-llama2-7b-v5p-8(test_llama2_7b.sh).llama_mistral_mixtral_orbax_to_hfstepAttributeError: No attribute 'params' in Stateat the first weightEXIT_CODE=1Logs (Cloud Logging, project
cloud-tpu-multipod-dev):Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.