Skip to content

fix(rl): add the local_layers logical-axis rule to rl.yml - #5987

Closed
chiajunglien wants to merge 1 commit into
mainfrom
emma/fix-rl-local-layers-axis-rule
Closed

chiajunglien wants to merge 1 commit into
mainfrom
emma/fix-rl-local-layers-axis-rule

Conversation

@chiajunglien

Copy link
Copy Markdown
Collaborator

Description

Since #5009, Qwen3.5 (like Qwen3-Next and Gemma4) stacks the linear-attention layers of each scanned block in a nested local_layers scan, so the RL actor's params carry a local_layers logical axis. post_train/rl.yml replaces base.yml's logical_axis_rules instead of extending them, and it was missing base.yml's ['local_layers', []] rule.

Maxtext places params with its own resolver, which leaves an axis without a rule unsharded, so checkpoint loading, rollout and logprobs are unaffected. Tunix, however, shards the trainer state from the metadata: PeftTrainer._shard_optimizer derives the shardings of the optimizer state and the gradient accumulator from nnx.get_partition_spec under the actor's rules, which keeps local_layers as is, and tunix.sft.sharding_utils.get_sharding falls back to a fully replicated sharding for any PartitionSpec that names an axis that is not on the mesh. For Qwen3.5-35B-A3B that is 94 GiB of float32 gradients (every decoder/layers/local_layers/* leaf, 30 GiB for each routed-expert kernel). JAX reshards a multi-device array to a replicated layout through x._value, so on Pathways each of these buffers is pulled into the head's host memory, and Step 2 (train_rl) of tests/end_to_end/tpu/qwen3.5/35b/test_qwen3.5_rl.sh is OOMKilled (100 GiB limit) before the first train step is traced. This has failed every DAG run since #5933 unblocked Step 1.

This adds the rule, in the same position as in base.yml.

DAG Error Log

Tests

  • New tests/post_training/unit/rl_logical_axis_rules_test.py: builds a scanned toy Qwen3.5 actor from rl.yml and checks that every axis in nnx.get_partition_spec under rl.yml's rules is a mesh axis. It fails on main with {'local_layers': 'decoder/layers/local_layers/attention/in_proj_ba/kernel'} and passes with this change.
  • Reproduced Tunix's _shard_optimizer sharding decision for the full-size Qwen3.5-35B-A3B RL actor on a 16-device trainer mesh (CPU): on main, 94.2 GiB of the 129.1 GiB float32 gradient buffers are resharded to a fully replicated layout; with this change, 0 GiB (the same as before Update Qwen3.5 scan-remat pattern by re-using Qwen3-Next's nested block scan #5009).
  • pyink, pylint, yamllint and codespell are clean; tests/unit/configs_test.py -k rl passes.
  • E2E on v5p-64 (Pathways, submitted with gcluster with the same flags as the maxtext_e2e_tpu_post_training DAG, on the image of a run that was OOMKilled, with only this rule added): all three steps of test_qwen3.5_rl.sh finish with Exit code: 0. The first train step compiles and runs in 79 s (then ~4.4 s per step), the step-2 checkpoint is saved and Step 3 decodes from it, and the head's workload-container peaks at 8.5 GiB.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @chiajunglien, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: No significant issues found.

Inferred spec & validation
  • Sources: PR description, code before/after.
  • Problem: Tunix shards optimizer state based on nnx.get_partition_spec under the actor's rules; missing ['local_layers', []] in rl.yml leaves the axis unsharded and resharded to replicated layout, pulling 94 GiB of gradients to Pathways head and causing an OOM.
Spec item Implementation Validating test
R1 (stated): Add local_layers rule to logical_axis_rules in rl.yml ✅ rl_logical_axis_rules_test.py::test_scanned_qwen3_5_param_axes_resolve_to_mesh_axes ✅
C1 (inferred): Prevent optimizer state gradients for local_layers parameters from replicating and causing OOM ✅ preserved rl_logical_axis_rules_test.py::test_scanned_qwen3_5_param_axes_resolve_to_mesh_axes ✅

@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses an Out-Of-Memory (OOM) issue occurring during RL training for Qwen3.5 models. The root cause was a missing 'local_layers' logical-axis rule in the RL configuration, which caused the trainer to fall back to a fully replicated sharding strategy for large gradient buffers. By adding the missing rule, the trainer now correctly shards these parameters, significantly reducing host memory usage and enabling successful training runs.

Highlights

  • Configuration Update: Added the 'local_layers' logical-axis rule to 'src/maxtext/configs/post_train/rl.yml' to ensure proper sharding for Qwen3.5 models.
  • New Regression Test: Introduced 'tests/post_training/unit/rl_logical_axis_rules_test.py' to verify that all model parameter axes resolve correctly to mesh axes, preventing future OOM issues.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

rl.yml replaces base.yml's logical_axis_rules instead of extending it, and it
was missing base.yml's `['local_layers', []]` rule. Since #5009, Qwen3.5 (like
Qwen3-Next and Gemma4) stacks the linear-attention layers of each scanned block
in a nested `local_layers` scan, so the actor's params carry that logical axis.

MaxText's own resolver leaves an axis without a rule unsharded, so the params are
placed as before. Tunix's PeftTrainer._shard_optimizer, however, derives the
shardings of the optimizer state and the gradient accumulator from
nnx.get_partition_spec under these rules, which keeps `local_layers` as is, and
tunix.sft.sharding_utils.get_sharding replicates any buffer whose PartitionSpec
names an axis that is not on the mesh. For Qwen3.5-35B-A3B that replicated
94 GiB of float32 gradients right before the first train step, and the
Pathways head (100 GiB limit) was OOMKilled.

Add the rule, and a CPU test that every axis of a scanned Qwen3.5 actor resolves
to mesh axes under rl.yml's rules.
@chiajunglien
chiajunglien force-pushed the emma/fix-rl-local-layers-axis-rule branch from a6a001b to d8b7ef9 Compare October 27, 2026 05:37
@codecov

codecov Bot commented Oct 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

copybara-service Bot pushed a commit that referenced this pull request Oct 27, 2026
FUTURE_COPYBARA_INTEGRATE_REVIEW=#5987 from AI-Hypercomputer:emma/fix-rl-local-layers-axis-rule d8b7ef9
PiperOrigin-RevId: 1008238768
copybara-service Bot pushed a commit that referenced this pull request Oct 27, 2026
FUTURE_COPYBARA_INTEGRATE_REVIEW=#5987 from AI-Hypercomputer:emma/fix-rl-local-layers-axis-rule d8b7ef9
PiperOrigin-RevId: 1008238768
copybara-service Bot pushed a commit that referenced this pull request Oct 27, 2026
FUTURE_COPYBARA_INTEGRATE_REVIEW=#5987 from AI-Hypercomputer:emma/fix-rl-local-layers-axis-rule d8b7ef9
PiperOrigin-RevId: 1008238768
copybara-service Bot pushed a commit that referenced this pull request Oct 27, 2026
…al-layers-axis-rule d8b7ef9

PiperOrigin-RevId: 1008255052
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants