Repository navigation
fix(rl): add the local_layers logical-axis rule to rl.yml - #5987
chiajunglien wants to merge 1 commit into
Conversation
|
🤖 Hi @chiajunglien, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
There was a problem hiding this comment.
📋 Review Summary
Verdict: No significant issues found.
Inferred spec & validation
- Sources: PR description, code before/after.
- Problem: Tunix shards optimizer state based on
nnx.get_partition_specunder the actor's rules; missing['local_layers', []]inrl.ymlleaves the axis unsharded and resharded to replicated layout, pulling 94 GiB of gradients to Pathways head and causing an OOM.
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): Add local_layers rule to logical_axis_rules in rl.yml |
✅ | rl_logical_axis_rules_test.py::test_scanned_qwen3_5_param_axes_resolve_to_mesh_axes ✅ |
C1 (inferred): Prevent optimizer state gradients for local_layers parameters from replicating and causing OOM |
✅ preserved | rl_logical_axis_rules_test.py::test_scanned_qwen3_5_param_axes_resolve_to_mesh_axes ✅ |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses an Out-Of-Memory (OOM) issue occurring during RL training for Qwen3.5 models. The root cause was a missing 'local_layers' logical-axis rule in the RL configuration, which caused the trainer to fall back to a fully replicated sharding strategy for large gradient buffers. By adding the missing rule, the trainer now correctly shards these parameters, significantly reducing host memory usage and enabling successful training runs. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
rl.yml replaces base.yml's logical_axis_rules instead of extending it, and it was missing base.yml's `['local_layers', []]` rule. Since #5009, Qwen3.5 (like Qwen3-Next and Gemma4) stacks the linear-attention layers of each scanned block in a nested `local_layers` scan, so the actor's params carry that logical axis. MaxText's own resolver leaves an axis without a rule unsharded, so the params are placed as before. Tunix's PeftTrainer._shard_optimizer, however, derives the shardings of the optimizer state and the gradient accumulator from nnx.get_partition_spec under these rules, which keeps `local_layers` as is, and tunix.sft.sharding_utils.get_sharding replicates any buffer whose PartitionSpec names an axis that is not on the mesh. For Qwen3.5-35B-A3B that replicated 94 GiB of float32 gradients right before the first train step, and the Pathways head (100 GiB limit) was OOMKilled. Add the rule, and a CPU test that every axis of a scanned Qwen3.5 actor resolves to mesh axes under rl.yml's rules.
a6a001b to
d8b7ef9
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…al-layers-axis-rule d8b7ef9 PiperOrigin-RevId: 1008255052
Description
Since #5009, Qwen3.5 (like Qwen3-Next and Gemma4) stacks the linear-attention layers of each scanned block in a nested
local_layersscan, so the RL actor's params carry alocal_layerslogical axis.post_train/rl.ymlreplaces base.yml'slogical_axis_rulesinstead of extending them, and it was missing base.yml's['local_layers', []]rule.Maxtext places params with its own resolver, which leaves an axis without a rule unsharded, so checkpoint loading, rollout and logprobs are unaffected. Tunix, however, shards the trainer state from the metadata:
PeftTrainer._shard_optimizerderives the shardings of the optimizer state and the gradient accumulator fromnnx.get_partition_specunder the actor's rules, which keepslocal_layersas is, andtunix.sft.sharding_utils.get_shardingfalls back to a fully replicated sharding for any PartitionSpec that names an axis that is not on the mesh. For Qwen3.5-35B-A3B that is 94 GiB of float32 gradients (everydecoder/layers/local_layers/*leaf, 30 GiB for each routed-expert kernel). JAX reshards a multi-device array to a replicated layout throughx._value, so on Pathways each of these buffers is pulled into the head's host memory, and Step 2 (train_rl) oftests/end_to_end/tpu/qwen3.5/35b/test_qwen3.5_rl.shis OOMKilled (100 GiB limit) before the first train step is traced. This has failed every DAG run since #5933 unblocked Step 1.This adds the rule, in the same position as in base.yml.
DAG Error Log
Tests
tests/post_training/unit/rl_logical_axis_rules_test.py: builds a scanned toy Qwen3.5 actor from rl.yml and checks that every axis innnx.get_partition_specunder rl.yml's rules is a mesh axis. It fails onmainwith{'local_layers': 'decoder/layers/local_layers/attention/in_proj_ba/kernel'}and passes with this change._shard_optimizersharding decision for the full-size Qwen3.5-35B-A3B RL actor on a 16-device trainer mesh (CPU): onmain, 94.2 GiB of the 129.1 GiB float32 gradient buffers are resharded to a fully replicated layout; with this change, 0 GiB (the same as before Update Qwen3.5 scan-remat pattern by re-using Qwen3-Next's nested block scan #5009).pyink,pylint,yamllintandcodespellare clean;tests/unit/configs_test.py -k rlpasses.gclusterwith the same flags as themaxtext_e2e_tpu_post_trainingDAG, on the image of a run that was OOMKilled, with only this rule added): all three steps of test_qwen3.5_rl.sh finish withExit code: 0. The first train step compiles and runs in 79 s (then ~4.4 s per step), the step-2 checkpoint is saved and Step 3 decodes from it, and the head'sworkload-containerpeaks at 8.5 GiB.Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.