Repository navigation
feat(testing): adapt the E2E pre-/post-training scripts to TPU v6e - #5990
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
2a5643b to
a586a6b
Compare
|
🤖 Hi @chiajunglien, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request migrates the existing E2E pre- and post-training test suite from TPU v5p to TPU v6e. Due to the smaller HBM capacity and different 2D topology of v6e chips, several scripts required configuration tuning to maintain performance and stability. The changes include switching to bfloat16 for decoding, adjusting expert and data parallelism, and enabling specific flags like layer scanning and physical axis splitting to ensure the workloads fit within the new hardware constraints. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
There was a problem hiding this comment.
📋 Review Summary
Verdict: No significant issues found.
Inferred spec & validation
- Sources: PR description, PR diff for the 10 modified test scripts.
- Problem: E2E test scripts for 10 models (Gemma3, Gemma4, Qwen3, GPT-OSS, Llama3.1) fail with OOM or topology errors on TPU v6e due to its smaller 32 GiB HBM (compared to v5p's 95 GiB) and 2D topology.
| Spec item | Implementation | Validating test |
|---|---|---|
| R1 (stated): Update 10 test scripts to fit on v6e slices (v6e-4 to v6e-64) | ✅ | All modified scripts ran end-to-end on v6e ✅ |
R2 (stated): Apply weight_dtype=bfloat16, opt_type=sgd, and adjust parallelism to fit in 32 GiB |
✅ | All modified scripts ran end-to-end on v6e ✅ |
R3 (stated): Add allow_split_physical_axes=True where EP/TP splits physical axes |
✅ | All modified scripts ran end-to-end on v6e ✅ |
| C1 (inferred): Ensure bash syntax and parameter interpolations are preserved | ✅ preserved | All modified scripts ran end-to-end on v6e ✅ |
The E2E tests are moving from v5p to v6e. A v6e chip has 32 GiB of HBM (v5p: 95 GiB) and v6e slices are 2D, so several scripts no longer fit in memory or cannot build their meshes as written. Write the v6e settings into those scripts: - vllm_decode: weight_dtype=bfloat16 (vllm_decode loads fp32 weights by default) and parallelism that matches the v6e slice sizes. - Training: more expert parallelism for the MoE models, allow_split_physical_axes=True where EP/TP must split the 2D topology, bfloat16 weights where fp32 does not fit, and Gemma4 pre-training from the scanned checkpoint. - RL: larger rollout TP, trainer meshes sized for half of a v6e-64 job, opt_type=sgd for Gemma4 and Llama, and rl.reshard_chunk_size=1 for Gemma4. - Qwen3.5 RL on v6e-64: an EP8 x FSDP2 x TP2 trainer and EP8 x TP2 rollouts. TP-only rollouts pad the MoE intermediate and do not fit, and TP x EP stays at 16 because vLLM pads the 2 KV heads to TP x EP, which must not exceed the 16 query heads. Every changed script was run end to end on v6e. After this change the scripts are sized for v6e and are not expected to fit on v5p as is.
a586a6b to
34abe0a
Compare
Move the MaxText E2E pre-training and post-training DAGs from bodaborg-v5p-nap to bodaborg-v6e-nap. Each test uses the v6e slice size that its MaxText script is sized for (AI-Hypercomputer/maxtext#5990). - Add TPU_V6E_BODABORG_NAP_CLUSTER (cloud-tpu-shared-capacity, southamerica-west1, namespace ubench-regression-tests) and Region.SOUTHAMERICA_WEST1. - Pre-training: gemma3-4b on v6e-4; gemma4-26b, qwen3-30b and gpt-oss-20b on v6e-32; llama3.1-70b on v6e-64. - Post-training: gemma3-4b and qwen3-vl-2b on v6e-4 (gemma3-4b RL on v6e-16); qwen3-30b SFT on v6e-16; gemma4-26b, llama3.1-70b and gpt-oss-20b SFT on v6e-32; gpt-oss-20b RL on v6e-32; qwen3-30b, gemma4-26b, llama3.1-70b and qwen3.5-35b RL on v6e-64. - gcluster: bodaborg-v6e-nap has both ct6e-standard-4t and ct6e-standard-8t node pools, so gcluster cannot resolve the "v6e" shorthand ("multiple matching machine types found in cluster for shorthand "v6e""). Multi-host v6e slices run only on ct6e-standard-4t hosts, so for v6e-16 and larger request ct6e-standard-4t with the slice topology. v6e-4 and v6e-8 keep the shorthand, which gcluster maps to a single machine type.
Move the MaxText E2E pre-training and post-training DAGs from bodaborg-v5p-nap to bodaborg-v6e-nap. Each test uses the v6e slice size that its MaxText script is sized for (AI-Hypercomputer/maxtext#5990). - Add TPU_V6E_BODABORG_NAP_CLUSTER (cloud-tpu-shared-capacity, southamerica-west1, namespace ubench-regression-tests) and Region.SOUTHAMERICA_WEST1. - Pre-training: gemma3-4b on v6e-4; gemma4-26b, qwen3-30b and gpt-oss-20b on v6e-32; llama3.1-70b on v6e-64. - Post-training: gemma3-4b and qwen3-vl-2b on v6e-4 (gemma3-4b RL on v6e-16); qwen3-30b SFT on v6e-16; gemma4-26b, llama3.1-70b and gpt-oss-20b SFT on v6e-32; gpt-oss-20b RL on v6e-32; qwen3-30b, gemma4-26b, llama3.1-70b and qwen3.5-35b RL on v6e-64. - gcluster: bodaborg-v6e-nap has both ct6e-standard-4t and ct6e-standard-8t node pools, so gcluster cannot resolve the "v6e" shorthand ("multiple matching machine types found in cluster for shorthand "v6e""). Multi-host v6e slices run only on ct6e-standard-4t hosts, so for v6e-16 and larger request ct6e-standard-4t with the slice topology. v6e-4 and v6e-8 keep the shorthand, which gcluster maps to a single machine type.
Move the MaxText E2E pre-training and post-training DAGs from bodaborg-v5p-nap to bodaborg-v6e-nap. Each test uses the v6e slice size that its MaxText script is sized for (AI-Hypercomputer/maxtext#5990). - Add TPU_V6E_BODABORG_NAP_CLUSTER (cloud-tpu-shared-capacity, southamerica-west1, namespace ubench-regression-tests) and Region.SOUTHAMERICA_WEST1. - Pre-training: gemma3-4b on v6e-4; gemma4-26b, qwen3-30b and gpt-oss-20b on v6e-32; llama3.1-70b on v6e-64. - Post-training: gemma3-4b and qwen3-vl-2b on v6e-4 (gemma3-4b RL on v6e-16); qwen3-30b SFT on v6e-16; gemma4-26b, llama3.1-70b and gpt-oss-20b SFT on v6e-32; gpt-oss-20b RL on v6e-32; qwen3-30b, gemma4-26b, llama3.1-70b and qwen3.5-35b RL on v6e-64. - gcluster: bodaborg-v6e-nap has both ct6e-standard-4t and ct6e-standard-8t node pools, so gcluster cannot resolve the "v6e" shorthand ("multiple matching machine types found in cluster for shorthand "v6e""). Multi-host v6e slices run only on ct6e-standard-4t hosts, so for v6e-16 and larger request ct6e-standard-4t with the slice topology. v6e-4 and v6e-8 keep the shorthand, which gcluster maps to a single machine type. - Wait up to 3 hours, instead of the default 60 minutes, for each TPU training job to start. The v6e queue only has TPU quota in a daily 4-hour window that the E2E suites share with other tests, so a job can queue for more than an hour. get_gke_config gets an optional workload_provision_timeout argument for this; its default is unchanged.
Move the MaxText E2E pre-training and post-training DAGs from bodaborg-v5p-nap to bodaborg-v6e-nap. Each test uses the v6e slice size that its MaxText script is sized for (AI-Hypercomputer/maxtext#5990). - Add TPU_V6E_BODABORG_NAP_CLUSTER (cloud-tpu-shared-capacity, southamerica-west1, namespace ubench-regression-tests) and Region.SOUTHAMERICA_WEST1. - Pre-training: gemma3-4b on v6e-4; gemma4-26b, qwen3-30b and gpt-oss-20b on v6e-32; llama3.1-70b on v6e-64. - Post-training: gemma3-4b and qwen3-vl-2b on v6e-4 (gemma3-4b RL on v6e-16); qwen3-30b SFT on v6e-16; gemma4-26b, llama3.1-70b and gpt-oss-20b SFT on v6e-32; gpt-oss-20b RL on v6e-32; qwen3-30b, gemma4-26b, llama3.1-70b and qwen3.5-35b RL on v6e-64. - gcluster: bodaborg-v6e-nap has both ct6e-standard-4t and ct6e-standard-8t node pools, so gcluster cannot resolve the "v6e" shorthand ("multiple matching machine types found in cluster for shorthand "v6e""). Multi-host v6e slices run only on ct6e-standard-4t hosts, so for v6e-16 and larger request ct6e-standard-4t with the slice topology. v6e-4 and v6e-8 keep the shorthand, which gcluster maps to a single machine type. - Wait up to 3 hours, instead of the default 60 minutes, for each TPU training job to start. The v6e queue only has TPU quota in a daily 4-hour window that the E2E suites share with other tests, so a job can queue for more than an hour. get_gke_config gets an optional workload_provision_timeout argument for this; its default is unchanged.
Move the MaxText E2E pre-training and post-training DAGs from bodaborg-v5p-nap to bodaborg-v6e-nap. Each test uses the v6e slice size that its MaxText script is sized for (AI-Hypercomputer/maxtext#5990). - Add TPU_V6E_BODABORG_NAP_CLUSTER (cloud-tpu-shared-capacity, southamerica-west1, namespace ubench-regression-tests) and Region.SOUTHAMERICA_WEST1. - Pre-training: gemma3-4b on v6e-4; gemma4-26b, qwen3-30b and gpt-oss-20b on v6e-32; llama3.1-70b on v6e-64. - Post-training: gemma3-4b and qwen3-vl-2b on v6e-4 (gemma3-4b RL on v6e-16); qwen3-30b SFT on v6e-16; gemma4-26b, llama3.1-70b and gpt-oss-20b SFT on v6e-32; gpt-oss-20b RL on v6e-32; qwen3-30b, gemma4-26b, llama3.1-70b and qwen3.5-35b RL on v6e-64. - gcluster: bodaborg-v6e-nap has both ct6e-standard-4t and ct6e-standard-8t node pools, so gcluster cannot resolve the "v6e" shorthand ("multiple matching machine types found in cluster for shorthand "v6e""). Multi-host v6e slices run only on ct6e-standard-4t hosts, so for v6e-16 and larger request ct6e-standard-4t with the slice topology. v6e-4 and v6e-8 keep the shorthand, which gcluster maps to a single machine type. - Wait up to 3 hours, instead of the default 60 minutes, for each TPU training job to start. The v6e queue only has TPU quota in a daily 4-hour window that the E2E suites share with other tests, so a job can queue for more than an hour. get_gke_config gets an optional workload_provision_timeout argument for this; its default is unchanged.
Move the MaxText E2E pre-training and post-training DAGs from bodaborg-v5p-nap to bodaborg-v6e-nap. Each test uses the v6e slice size that its MaxText script is sized for (AI-Hypercomputer/maxtext#5990). - Add TPU_V6E_BODABORG_NAP_CLUSTER (cloud-tpu-shared-capacity, southamerica-west1, namespace ubench-regression-tests) and Region.SOUTHAMERICA_WEST1. - Pre-training: gemma3-4b on v6e-4; gemma4-26b, qwen3-30b and gpt-oss-20b on v6e-32; llama3.1-70b on v6e-64. - Post-training: gemma3-4b and qwen3-vl-2b on v6e-4 (gemma3-4b RL on v6e-16); qwen3-30b SFT on v6e-16; gemma4-26b, llama3.1-70b and gpt-oss-20b SFT on v6e-32; gpt-oss-20b RL on v6e-32; qwen3-30b, gemma4-26b, llama3.1-70b and qwen3.5-35b RL on v6e-64. - gcluster: bodaborg-v6e-nap has both ct6e-standard-4t and ct6e-standard-8t node pools, so gcluster cannot resolve the "v6e" shorthand ("multiple matching machine types found in cluster for shorthand "v6e""). Multi-host v6e slices run only on ct6e-standard-4t hosts, so for v6e-16 and larger request ct6e-standard-4t with the slice topology. v6e-4 and v6e-8 keep the shorthand, which gcluster maps to a single machine type. - Wait up to 3 hours, instead of the default 60 minutes, for each TPU training job to start. The v6e queue only has TPU quota in a daily 4-hour window that the E2E suites share with other tests, so a job can queue for more than an hour. get_gke_config gets an optional workload_provision_timeout argument for this; its default is unchanged.
…o-HF flags Bring the to-HF changes from release/384 (GoogleCloudPlatform#1435) into the v6e migration, and fix the Gemma4 pre-training to-HF flags. - Wait up to 3 hours for the to-HF conversion jobs on the m1-megamem-96-shared CPU cluster to start. That cluster has two m1-megamem-96 nodes and admits two conversion jobs at a time, while one E2E run submits ten conversion jobs there: on 2026-11-02 the llama3.1-70b to-HF job was still queued after 60 minutes, so wait_for_workload_start timed out. get_maxtext_cpu_end_to_end_gke_config gets an optional workload_provision_timeout argument. The to-HF tasks on the n2-highmem cluster keep the 60-minute default. - Convert the gemma4-26b pre-training checkpoint to Hugging Face with scan_layers=true. Since AI-Hypercomputer/maxtext#5990 the Gemma4 pre-training script trains from the scanned checkpoint, so the to-HF step failed with "Your run specifies scan_layers=False, but the checkpoint was saved with scan_layers=True".
…o-HF flags The MaxText E2E to-HF conversions for the large models run on the m1-megamem-96-shared CPU cluster after training. That cluster has two m1-megamem-96 nodes, and its Kueue ClusterQueue admits two conversion jobs at a time, while one E2E run submits ten conversion jobs there. Later jobs wait longer and longer: on 2026-11-02 the llama3.1-70b to-HF job was still queued after 60 minutes, so wait_for_workload_start timed out and the task failed. - get_maxtext_cpu_end_to_end_gke_config gets an optional workload_provision_timeout argument. Its default is unchanged, so the other DAGs that use it are not affected. - In the pre-training and post-training E2E DAGs, the to-HF tasks that run on the m1-megamem-96 cluster wait up to 3 hours for their job to start. The to-HF tasks on the n2-highmem cluster keep the 60-minute default, so a job stuck in Pending there still fails after an hour. - Convert the gemma4-26b pre-training checkpoint to Hugging Face with scan_layers=true. Since AI-Hypercomputer/maxtext#5990 the Gemma4 pre-training script trains from the scanned checkpoint, so the to-HF step failed with "Your run specifies scan_layers=False, but the checkpoint was saved with scan_layers=True".
…o-HF flags Bring the to-HF changes from release/384 (GoogleCloudPlatform#1435) into the v6e migration, and fix the Gemma4 pre-training to-HF flags. - Wait up to 3 hours for the to-HF conversion jobs on the m1-megamem-96-shared CPU cluster to start. That cluster has two m1-megamem-96 nodes and admits two conversion jobs at a time, while one E2E run submits ten conversion jobs there: on 2026-11-02 the llama3.1-70b to-HF job was still queued after 60 minutes, so wait_for_workload_start timed out. get_maxtext_cpu_end_to_end_gke_config gets an optional workload_provision_timeout argument. The to-HF tasks on the n2-highmem cluster keep the 60-minute default. - Convert the gemma4-26b pre-training checkpoint to Hugging Face with scan_layers=true. Since AI-Hypercomputer/maxtext#5990 the Gemma4 pre-training script trains from the scanned checkpoint, so the to-HF step failed with "Your run specifies scan_layers=False, but the checkpoint was saved with scan_layers=True".
Move the MaxText E2E TPU pre-training and post-training DAGs from `bodaborg-v5p-nap` to `bodaborg-v6e-nap`. Each test runs on the v6e slice size its MaxText script is sized for in AI-Hypercomputer/maxtext#5990. This PR also includes the to-HF changes from #1435.
`maxtext_moe_tpu_e2e` runs `gpt-oss-20b` (`tests/end_to_end/tpu/gpt_oss/20b/test_gpt_oss_to_mt.sh` followed by `tests/end_to_end/tpu/gpt_oss/20b/test_gpt_oss.sh`) on `GkeClusters.TPU_V5P_BODABORG_NAP_CLUSTER` with default `core_count=8` (`v5p-8`, 1 slice = 4 TPU chips = 4 JAX devices). AI-Hypercomputer/maxtext#5990 adapted the E2E training scripts for TPU v6e (*"After this change the scripts are sized for v6e and are not expected to fit on v5p as is"*) and added `ici_expert_parallelism=8` to the `maxtext.trainers.pre_train.train` command in `tests/end_to_end/tpu/gpt_oss/20b/test_gpt_oss.sh` to size it for `v6e-32` (32 chips). While GoogleCloudPlatform#1428 migrated `gpt-oss-20b` in `maxtext_e2e_tpu_pre_training` to `v6e-32` on `bodaborg-v6e-nap`, `gpt-oss-20b` in `maxtext_moe_tpu_e2e` remained on `v5p-8` (4 devices). When `fill_unspecified_mesh_axes` attempts to resolve `ici_fsdp_parallelism=-1` across 4 devices with `ici_expert_parallelism=8` (`4 / 8 = 0.5`), both the stable and nightly tasks fail before the first training step (b/582148023): ``` + python3 -m maxtext.trainers.pre_train.train ... model_name=gpt-oss-20b scan_layers=false use_multimodal=false ici_expert_parallelism=8 ... File "/deps/src/maxtext/utils/maxtext_utils.py", line 2208, in create_device_mesh ici_parallelism = max_utils.fill_unspecified_mesh_axes(ici_parallelism, num_devices_per_slice, "ICI") File "/deps/src/maxtext/utils/max_utils.py", line 494, in fill_unspecified_mesh_axes determined_val >= 1 and determined_val.is_integer() AssertionError: Unspecified value unable to be determined with the given ICI parallelism values ``` As suggested by AI-Hypercomputer/maxtext#5990 (and matching GoogleCloudPlatform#1428), this PR moves `gpt-oss-20b` in `maxtext_moe_tpu_e2e` from `v5p-8` to `GkeClusters.TPU_V6E_BODABORG_NAP_CLUSTER.override(core_count=32)` (`v6e-32`) and adds `"v6e-32"` to the DAG tags.
`maxtext_moe_tpu_e2e` runs `gpt-oss-20b` (`tests/end_to_end/tpu/gpt_oss/20b/test_gpt_oss_to_mt.sh` followed by `tests/end_to_end/tpu/gpt_oss/20b/test_gpt_oss.sh`) on `GkeClusters.TPU_V5P_BODABORG_NAP_CLUSTER` with default `core_count=8` (`v5p-8`, 1 slice = 4 TPU chips = 4 JAX devices). AI-Hypercomputer/maxtext#5990 adapted the E2E training scripts for TPU v6e (*"After this change the scripts are sized for v6e and are not expected to fit on v5p as is"*) and added `ici_expert_parallelism=8` to the `maxtext.trainers.pre_train.train` command in `tests/end_to_end/tpu/gpt_oss/20b/test_gpt_oss.sh` to size it for `v6e-32` (32 chips). While GoogleCloudPlatform#1428 migrated `gpt-oss-20b` in `maxtext_e2e_tpu_pre_training` to `v6e-32` on `bodaborg-v6e-nap`, `gpt-oss-20b` in `maxtext_moe_tpu_e2e` remained on `v5p-8` (4 devices). When `fill_unspecified_mesh_axes` attempts to resolve `ici_fsdp_parallelism=-1` across 4 devices with `ici_expert_parallelism=8` (`4 / 8 = 0.5`), both the stable and nightly tasks fail before the first training step (b/582148023): ``` + python3 -m maxtext.trainers.pre_train.train ... model_name=gpt-oss-20b scan_layers=false use_multimodal=false ici_expert_parallelism=8 ... File "/deps/src/maxtext/utils/maxtext_utils.py", line 2208, in create_device_mesh ici_parallelism = max_utils.fill_unspecified_mesh_axes(ici_parallelism, num_devices_per_slice, "ICI") File "/deps/src/maxtext/utils/max_utils.py", line 494, in fill_unspecified_mesh_axes determined_val >= 1 and determined_val.is_integer() AssertionError: Unspecified value unable to be determined with the given ICI parallelism values ``` As suggested by AI-Hypercomputer/maxtext#5990 (and matching GoogleCloudPlatform#1428), this PR moves `gpt-oss-20b` in `maxtext_moe_tpu_e2e` from `v5p-8` to `GkeClusters.TPU_V6E_BODABORG_NAP_CLUSTER.override(core_count=32)` (`v6e-32`) and adds `"v6e-32"` to the DAG tags.
) `maxtext_moe_tpu_e2e` runs `gpt-oss-20b` (`tests/end_to_end/tpu/gpt_oss/20b/test_gpt_oss_to_mt.sh` followed by `tests/end_to_end/tpu/gpt_oss/20b/test_gpt_oss.sh`) on `GkeClusters.TPU_V5P_BODABORG_NAP_CLUSTER` with default `core_count=8` (`v5p-8`, 1 slice = 4 TPU chips = 4 JAX devices). AI-Hypercomputer/maxtext#5990 adapted the E2E training scripts for TPU v6e (*"After this change the scripts are sized for v6e and are not expected to fit on v5p as is"*) and added `ici_expert_parallelism=8` to the `maxtext.trainers.pre_train.train` command in `tests/end_to_end/tpu/gpt_oss/20b/test_gpt_oss.sh` to size it for `v6e-32` (32 chips). While #1428 migrated `gpt-oss-20b` in `maxtext_e2e_tpu_pre_training` to `v6e-32` on `bodaborg-v6e-nap`, `gpt-oss-20b` in `maxtext_moe_tpu_e2e` remained on `v5p-8` (4 devices). When `fill_unspecified_mesh_axes` attempts to resolve `ici_fsdp_parallelism=-1` across 4 devices with `ici_expert_parallelism=8` (`4 / 8 = 0.5`), both the stable and nightly tasks fail before the first training step (b/582148023): ``` + python3 -m maxtext.trainers.pre_train.train ... model_name=gpt-oss-20b scan_layers=false use_multimodal=false ici_expert_parallelism=8 ... File "/deps/src/maxtext/utils/maxtext_utils.py", line 2208, in create_device_mesh ici_parallelism = max_utils.fill_unspecified_mesh_axes(ici_parallelism, num_devices_per_slice, "ICI") File "/deps/src/maxtext/utils/max_utils.py", line 494, in fill_unspecified_mesh_axes determined_val >= 1 and determined_val.is_integer() AssertionError: Unspecified value unable to be determined with the given ICI parallelism values ``` As suggested by AI-Hypercomputer/maxtext#5990 (and matching #1428), this PR moves `gpt-oss-20b` in `maxtext_moe_tpu_e2e` from `v5p-8` to `GkeClusters.TPU_V6E_BODABORG_NAP_CLUSTER.override(core_count=32)` (`v6e-32`) and adds `"v6e-32"` to the DAG tags.
Description
The E2E pre-/post-training tests are moving from TPU v5p to TPU v6e. A v6e chip has 32 GiB of HBM (v5p: 95 GiB) and v6e slices have a 2D topology, so 11 of the scripts no longer fit in memory or cannot build their meshes as written. This PR writes the v6e settings into those scripts. Every changed script was run end to end on v6e.
Important
After this change, these scripts are sized for the v6e slices below and are not expected to fit on v5p as is. The DAG needs to switch these tests to v6e, with these slice sizes, when this lands.
Changes per script
gemma3/4b/test_gemma3_lora.shvllm_decodesteps:weight_dtype=bfloat16qwen3/30b/test_qwen3_sft.shvllm_decodesteps: bf16, EP 2→4, DP 2→1 (TP4 x EP4).train_sft:allow_split_physical_axes=Trueqwen3/30b/test_qwen3_rl.shvllm_decodesteps: bf16,hbm_utilization_vllm0.8→0.5, DP 2→8.train_rl: EP8 x FSDP2 x TP2 withallow_split_physical_axes=True, rollouthbm_utilization_vllm0.55→0.65gpt_oss/20b/test_gpt_oss.shici_expert_parallelism=8gemma4/26b/test_gemma4.shscan_layers=true, EP8, bf16. Step 3 decode:scan_layers=true, because the training checkpoint is now scannedgemma4/26b/test_gemma4_sft.shvllm_decodesteps: bf16.train_sft: EP8, bf16,opt_type=sgd,max_target_length=1024gemma4/26b/test_gemma4_rl.shvllm_decodesteps: bf16, TP 4→8, DP8.train_rl: EP 4→8, TP2,allow_split_physical_axes=True,opt_type=sgd,rl.num_generations=4,rl.reshard_chunk_size=1, rollout TP 2→4llama3.1/70b/test_llama3.1_70b.shallow_split_physical_axes=Truellama3.1/70b/test_llama3.1_70b_sft.shvllm_decodesteps: bf16. Step 3 decode: TP 8→16, DP 4→2.train_sft: bf16llama3.1/70b/test_llama3.1_70b_rl.shvllm_decodesteps: bf16. Step 1: DP 4→8. Step 3: TP 8→16.train_rl: rollout TP 4→8,opt_type=sgdqwen3.5/35b/test_qwen3.5_rl.shtrain_rl: EP8 x FSDP2 x TP2 withallow_split_physical_axes=True; rollout TP4 → EP8 x TP2These scripts pass on v6e unchanged:
gemma3-4b: pre-training, SFT and multimodal SFT on v6e-4; RL on v6e-16.qwen3-vl-2b: multimodal SFT on v6e-4.qwen3-30b: pre-training on v6e-32.gpt-oss-20b: SFT and RL on v6e-32.Why these changes
vllm_decodeloads fp32 weights by default; bf16 halves their memory. The RL rollouts already use bf16.allow_split_physical_axes=True.rl.reshard_chunk_size=1moves them in chunks.Changes to test content
opt_type=sgdinstead of the defaultadamw: Gemma4 SFT, Gemma4 RL and Llama RL. The Qwen3 scripts already use it.max_target_length=1024instead of the default 2048 for Gemma4 SFT.rl.num_generations2→4 for Gemma4 RL. The micro batch must dividefsdp x expert.Tests
Each changed script was run end to end on
bodaborg-v6e-napwith the slice size from the table:tpu_pre_training:1ed249c/tpu_post_training:1ed249c. At the time, the scripts in the image matchedmain.e2e-39297931486.The validation runs applied these settings to the scripts inside the container. They differ from this PR as follows:
-scanoutput directory, to avoid overwriting the v5p outputs.USE_MULTIMODAL=true) also getsscan_layers=truehere. The E2E test does not run that branch._fuse_moe_weightsfix from fix(vllm): use [gate | up] prefused MoE layout under expert parallelism #5988 applied, so its decodes are coherent withprefuse_moe_weights=True. Without fix(vllm): use [gate | up] prefused MoE layout under expert parallelism #5988 these decodes print gibberish, as they do on v5p today, but the script still passes.tpu_post_training:047d606image, checkpointe2e-20261028,debug=Trueand a separate output directory. On top of the image it applied the fix(vllm): use [gate | up] prefused MoE layout under expert parallelism #5988 fix and thelocal_layersrule from fix(rl): add the local_layers logical-axis rule to rl.yml #5987, which was merged after that image was built.wiin the tensor-parallel interleaved layout (MoEFusedLayout.PER_SHARD_INTERLEAVE); under EP the kernel expects[gate | up], as fixed for checkpoint loading in fix(vllm): use [gate | up] prefused MoE layout under expert parallelism #5988. The script still passes; the weight sync fix is a follow-up.Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.