Skip to content

fix(ckpt-conversion): port llama_mistral_mixtral_orbax_to_hf to the NNX training state - #6035

Closed
ryanliuCienet wants to merge 1 commit into
mainfrom
ryan/fix-orbax-to-hf-nnx-state
Closed

ryanliuCienet wants to merge 1 commit into
mainfrom
ryan/fix-orbax-to-hf-nnx-state

Conversation

@ryanliuCienet

@ryanliuCienet ryanliuCienet commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Description

b1ac62ab0 ("NNX: flip pure_nnx/enable_nnx/pure_nnx_decoder defaults to True", #3526) made _read_train_checkpoint -- which this standalone script borrows from generate_param_only_checkpoint -- return the NNX state by default: a flat nnx.State with the weights under model and the optimizer under optimizer, instead of a Linen TrainState. bbb843f1d ("[NNX] Delete Linen (5/5): remove the pure_nnx/enable_nnx/pure_nnx_decoder config flags", #4361) then removed the Linen path altogether. llama_mistral_mixtral_orbax_to_hf was never ported and still indexes the old layout, training_state.params["params"][...], so every conversion dies on the first weight it touches:

AttributeError: No attribute 'params' in State

In tests/end_to_end/tpu/llama2/7b/test_llama2_7b.sh this is the MaxText -> HF step after the first logit check. It has never been visible in prod: the script fails earlier, at the logit checker's tokenizer load (b/579251787, #6025) and, before that, at the generate_param_only_checkpoint PRNG mismatch (b/576849994, #6026). It is the next failure once those two are in. tests/end_to_end/tpu/llama3.1/8b/{2,3}_test_llama3.1_8b.sh run the same script.

Changes

src/maxtext/checkpoint_conversion/standalone_scripts/llama_mistral_mixtral_orbax_to_hf.py only:

  • load_model_state -> load_model_params: reshape the loaded state once with train_state_nnx.to_checkpoint_dict -- the helper the checkpoint writer uses, so the result is exactly the on-disk Linen layout -- and return its ["params"]["params"] subtree, i.e. what used to be TrainState.params["params"]. convert_gpt3_ckpt_from_paxml already loads its state the same way.
  • convert_state_to_hf(params, ...): the 16 training_state.params["params"][...] lookups become params[...]; the porting logic itself is unchanged.

BUGS: b/579252268

Related: b/576849994 and b/579251787 (the PRNG and tokenizer failures that kept this step unreachable; fixed in #6026 and #6025).

Tests

CPU check

Stable image maxtext_jax_stable:2026-10-27, JAX_PLATFORMS=cpu, pylint 10.00/10 and pyink --check clean.

Tiny llama2 (dense) and mixtral (MoE, 2 experts) models built through load_model_params -- a fresh init is the same nnx.State the restore path returns -- then convert_state_to_hf with matching MODEL_PARAMS_DICT entries: every HF tensor name and shape as expected (21 dense / 29 MoE tensors), values equal to the MaxText arrays (embed_tokens, v_proj, down_proj, MoE w1 spot-checked). The pre-fix script fails on the same models with the AttributeError above.

E2E on TPU v5p-8

maxtext_end_to_end DAG (dev Composer env), cluster auto-v5p-8-bodaborg, image maxtext_jax_stable:2026-10-27 + this branch, #6024, #6025 and #6026 (maxtext_testing_liuryan:testing_all-b22ff6314), task maxtext-stable-llama2-7b-v5p-8 (test_llama2_7b.sh).

Image llama_mistral_mixtral_orbax_to_hf step Script
Prod image + #6025 + #6026 (before this PR) AttributeError: No attribute 'params' in State at the first weight EXIT_CODE=1
This PR + #6024/#6025/#6026 pending (fill in: conversion result and the HF-checkpoint logit check that follows) pending

Logs (Cloud Logging, project cloud-tpu-multipod-dev):

  • Before: llama2-7b
  • This PR: pending (fill in workload link)

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

# Description

Root cause: #b1ac62ab0 ("NNX: flip pure_nnx/enable_nnx/pure_nnx_decoder
defaults to True") made `_read_train_checkpoint` -- which this standalone
script borrows from `generate_param_only_checkpoint` -- return the NNX state by
default: a flat `nnx.State` with the weights under `model` and the optimizer
under `optimizer`, instead of a Linen `TrainState`. #bbb843f1d ("[NNX] Delete
Linen (5/5)") then removed the Linen path altogether. The script was never
ported and still indexes the old layout, `training_state.params["params"]`, so
every conversion dies on the first weight it touches:

    AttributeError: No attribute 'params' in State

In `tests/end_to_end/tpu/llama2/7b/test_llama2_7b.sh` this is the MaxText -> HF
step after the first logit check; it stayed hidden behind the earlier failures
in that script (the logit checker tokenizer, #<TOKENIZER_PR_NUMBER>, and the
`generate_param_only_checkpoint` PRNG mismatch, #<PRNG_PR_NUMBER>) and is only
reachable once those are in. `tests/end_to_end/tpu/llama3.1/8b/{2,3}_test_llama3.1_8b.sh`
run the same script.

Reshape the loaded state once with `train_state_nnx.to_checkpoint_dict` -- the
helper the checkpoint writer uses, so the result is exactly the on-disk Linen
layout -- and hand `convert_state_to_hf` the `["params"]["params"]` subtree it
was written against. The porting code is unchanged apart from the shorter
indexing; `convert_gpt3_ckpt_from_paxml` already loads its state the same way.

# Tests

- `pylint` 10.00/10, `pyink --check` clean.
- CPU (`JAX_PLATFORMS=cpu`, stable image): tiny llama2 (dense) and mixtral
  (MoE, 2 experts) models built through `load_model_params` -- a fresh init is
  the same `nnx.State` the restore path returns -- then `convert_state_to_hf`
  with matching `MODEL_PARAMS_DICT` entries: every HF tensor name and shape as
  expected, values equal to the MaxText arrays. The pre-fix script fails on the
  same models with the `AttributeError` above.
- E2E on v5p-8 via the `maxtext_end_to_end` DAG with a test image built from
  this branch: pending (fill in workload link for llama2-7b).

# Checklist

Before submitting this PR, please make sure (put X in square brackets):
- [X] I have performed a self-review of my code. For an optional AI review, add the `gemini-review` label.
- [X] I have necessary comments in my code, particularly in hard-to-understand areas.
- [X] I have run end-to-end tests and provided workload links above if applicable.
- [X] I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in [our documentation](https://maxtext.readthedocs.io/en/latest/development.html#adding-new-documentation-files).
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @ryanliuCienet, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request updates the MaxText to HuggingFace checkpoint conversion script to support the transition from Linen-based training states to the new NNX-based state. By reshaping the loaded NNX state into the legacy Linen layout, the conversion logic is restored to functionality without requiring significant changes to the downstream parameter mapping.

Highlights

  • NNX Checkpoint Compatibility: Updated the checkpoint conversion script to handle the new NNX-based training state layout, resolving an AttributeError encountered during conversion.
  • Refactored Parameter Loading: Renamed load_model_state to load_model_params and updated it to return the model weights in the expected Linen layout using train_state_nnx.to_checkpoint_dict.
  • Simplified Parameter Access: Updated convert_state_to_hf to accept the flattened parameter dictionary, removing redundant indexing into the old training_state structure.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: No significant issues found.

Inferred spec & validation
  • Sources: PR description, code before/after.
  • Problem: _read_train_checkpoint now returns an NNX State object rather than a Linen TrainState, causing llama_mistral_mixtral_orbax_to_hf to crash with AttributeError when looking up the params attribute.
Spec item Implementation Validating test
R1 (stated): Conversion script extracts weights successfully from the NNX state. ✅ test_llama2_7b.sh ✅
R2 (stated): Extracted weights map to the HuggingFace layout correctly without dimension errors. ✅ test_llama2_7b.sh ✅
C1 (inferred): Existing conversion logic for parameters remains unchanged. ✅ preserved test_llama2_7b.sh ✅

@ryanliuCienet ryanliuCienet changed the title Port llama_mistral_mixtral_orbax_to_hf to the NNX training state fix(ckpt-conversion): port llama_mistral_mixtral_orbax_to_hf to the NNX training state Oct 29, 2026
@codecov

codecov Bot commented Oct 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-ci-tests Add this label to the PR to skip running CI tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant