Skip to content

feat(testing): adapt the E2E pre-/post-training scripts to TPU v6e - #5990

Merged
copybara-service[bot] merged 1 commit into
mainfrom
emma/e2e-scripts-v6e
Nov 2, 2026
Merged

copybara-service[bot] merged 1 commit into
mainfrom
emma/e2e-scripts-v6e

Conversation

@chiajunglien

@chiajunglien chiajunglien commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Description

The E2E pre-/post-training tests are moving from TPU v5p to TPU v6e. A v6e chip has 32 GiB of HBM (v5p: 95 GiB) and v6e slices have a 2D topology, so 11 of the scripts no longer fit in memory or cannot build their meshes as written. This PR writes the v6e settings into those scripts. Every changed script was run end to end on v6e.

Important

After this change, these scripts are sized for the v6e slices below and are not expected to fit on v5p as is. The DAG needs to switch these tests to v6e, with these slice sizes, when this lands.

Changes per script

Script v6e slice Changes
gemma3/4b/test_gemma3_lora.sh v6e-4 Both vllm_decode steps: weight_dtype=bfloat16
qwen3/30b/test_qwen3_sft.sh v6e-16 Both vllm_decode steps: bf16, EP 2→4, DP 2→1 (TP4 x EP4). train_sft: allow_split_physical_axes=True
qwen3/30b/test_qwen3_rl.sh v6e-64 Both vllm_decode steps: bf16, hbm_utilization_vllm 0.8→0.5, DP 2→8. train_rl: EP8 x FSDP2 x TP2 with allow_split_physical_axes=True, rollout hbm_utilization_vllm 0.55→0.65
gpt_oss/20b/test_gpt_oss.sh v6e-32 Training: ici_expert_parallelism=8
gemma4/26b/test_gemma4.sh v6e-32 Training: scanned checkpoint with scan_layers=true, EP8, bf16. Step 3 decode: scan_layers=true, because the training checkpoint is now scanned
gemma4/26b/test_gemma4_sft.sh v6e-32 Both vllm_decode steps: bf16. train_sft: EP8, bf16, opt_type=sgd, max_target_length=1024
gemma4/26b/test_gemma4_rl.sh v6e-64 Both vllm_decode steps: bf16, TP 4→8, DP8. train_rl: EP 4→8, TP2, allow_split_physical_axes=True, opt_type=sgd, rl.num_generations=4, rl.reshard_chunk_size=1, rollout TP 2→4
llama3.1/70b/test_llama3.1_70b.sh v6e-64 Training: allow_split_physical_axes=True
llama3.1/70b/test_llama3.1_70b_sft.sh v6e-32 Both vllm_decode steps: bf16. Step 3 decode: TP 8→16, DP 4→2. train_sft: bf16
llama3.1/70b/test_llama3.1_70b_rl.sh v6e-64 Both vllm_decode steps: bf16. Step 1: DP 4→8. Step 3: TP 8→16. train_rl: rollout TP 4→8, opt_type=sgd
qwen3.5/35b/test_qwen3.5_rl.sh v6e-64 train_rl: EP8 x FSDP2 x TP2 with allow_split_physical_axes=True; rollout TP4 → EP8 x TP2

These scripts pass on v6e unchanged:

  • gemma3-4b: pre-training, SFT and multimodal SFT on v6e-4; RL on v6e-16.
  • qwen3-vl-2b: multimodal SFT on v6e-4.
  • qwen3-30b: pre-training on v6e-32.
  • gpt-oss-20b: SFT and RL on v6e-32.

Why these changes

  1. vllm_decode loads fp32 weights by default; bf16 halves their memory. The RL rollouts already use bf16.
  2. Splitting EP8 or TP4 out of a 2D v6e slice (4x4, 4x8, 8x8) needs allow_split_physical_axes=True.
  3. MoE training without layer scanning needs a lot of memory; more expert parallelism or a scanned checkpoint makes it fit.
  4. The RL weight sync needs an extra copy of the weights. rl.reshard_chunk_size=1 moves them in chunks.
  5. A 30B-class MoE RL trainer needs 32 v6e chips (half of a v6e-64 job). On v5p it used 8 chips with 95 GiB each.
  6. Qwen3.5 rollouts need expert parallelism: TP-only rollouts pad the MoE intermediate dimension and run out of HBM. TP x EP stays at 16 because vLLM pads the 2 KV heads to TP x EP, which must not exceed the 16 query heads.

Changes to test content

  • opt_type=sgd instead of the default adamw: Gemma4 SFT, Gemma4 RL and Llama RL. The Qwen3 scripts already use it.
  • max_target_length=1024 instead of the default 2048 for Gemma4 SFT.
  • rl.num_generations 2→4 for Gemma4 RL. The micro batch must divide fsdp x expert.
  • Gemma4 pre-training now starts from the scanned checkpoint, as the comment above that step already suggests.

Tests

Each changed script was run end to end on bodaborg-v6e-nap with the slice size from the table:

  • Image: tpu_pre_training:1ed249c / tpu_post_training:1ed249c. At the time, the scripts in the image matched main.
  • Checkpoints: e2e-39297931486.
  • Runtime: Pathways for post-training, matching the DAG.
Script Result Log
Gemma3 LoRA ✅ Both decodes coherent log
Qwen3 SFT ✅ log
Qwen3 RL ✅ Accuracy 25% → 37.5%, both decodes coherent log
GPT-OSS pre-training ✅ Loss 5.705 → 3.158 log
Gemma4 pre-training ✅ Loss 10.555 → 4.967; post-training decode matches the v5p run log
Gemma4 SFT ✅ Both decodes coherent log
Gemma4 RL ✅ 41 min, accuracy 87.5% → 87.5%, both decodes coherent log
Llama3.1 pre-training ✅ 47 min, loss 1.882 → 1.923 log
Llama3.1 SFT ✅ log
Llama3.1 RL ✅ 40 min, rollouts coherent, format reward 100% log
Qwen3.5 RL ✅ 35 min, both decodes coherent; RL rollouts incoherent (see below) log

The validation runs applied these settings to the scripts inside the container. They differ from this PR as follows:

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@codecov

codecov Bot commented Oct 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@chiajunglien
chiajunglien marked this pull request as ready for review October 28, 2026 01:57
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @chiajunglien, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request migrates the existing E2E pre- and post-training test suite from TPU v5p to TPU v6e. Due to the smaller HBM capacity and different 2D topology of v6e chips, several scripts required configuration tuning to maintain performance and stability. The changes include switching to bfloat16 for decoding, adjusting expert and data parallelism, and enabling specific flags like layer scanning and physical axis splitting to ensure the workloads fit within the new hardware constraints.

Highlights

  • TPU v6e Migration: Updated E2E pre- and post-training test scripts to support TPU v6e architecture, addressing memory constraints and topology requirements.
  • Configuration Adjustments: Optimized memory usage by enabling bfloat16 for vLLM decoding and adjusted parallelism settings (EP, TP, DP) across multiple models to fit within v6e slice constraints.
  • Training Improvements: Integrated layer scanning for Gemma4 and enabled 'allow_split_physical_axes' for MoE models to improve memory efficiency and compatibility with v6e hardware.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: No significant issues found.

Inferred spec & validation
  • Sources: PR description, PR diff for the 10 modified test scripts.
  • Problem: E2E test scripts for 10 models (Gemma3, Gemma4, Qwen3, GPT-OSS, Llama3.1) fail with OOM or topology errors on TPU v6e due to its smaller 32 GiB HBM (compared to v5p's 95 GiB) and 2D topology.
Spec item Implementation Validating test
R1 (stated): Update 10 test scripts to fit on v6e slices (v6e-4 to v6e-64) ✅ All modified scripts ran end-to-end on v6e ✅
R2 (stated): Apply weight_dtype=bfloat16, opt_type=sgd, and adjust parallelism to fit in 32 GiB ✅ All modified scripts ran end-to-end on v6e ✅
R3 (stated): Add allow_split_physical_axes=True where EP/TP splits physical axes ✅ All modified scripts ran end-to-end on v6e ✅
C1 (inferred): Ensure bash syntax and parameter interpolations are preserved ✅ preserved All modified scripts ran end-to-end on v6e ✅

The E2E tests are moving from v5p to v6e. A v6e chip has 32 GiB of HBM
(v5p: 95 GiB) and v6e slices are 2D, so several scripts no longer fit in
memory or cannot build their meshes as written. Write the v6e settings
into those scripts:

- vllm_decode: weight_dtype=bfloat16 (vllm_decode loads fp32 weights by
  default) and parallelism that matches the v6e slice sizes.
- Training: more expert parallelism for the MoE models,
  allow_split_physical_axes=True where EP/TP must split the 2D topology,
  bfloat16 weights where fp32 does not fit, and Gemma4 pre-training from
  the scanned checkpoint.
- RL: larger rollout TP, trainer meshes sized for half of a v6e-64 job,
  opt_type=sgd for Gemma4 and Llama, and rl.reshard_chunk_size=1 for
  Gemma4.
- Qwen3.5 RL on v6e-64: an EP8 x FSDP2 x TP2 trainer and EP8 x TP2
  rollouts. TP-only rollouts pad the MoE intermediate and do not fit, and
  TP x EP stays at 16 because vLLM pads the 2 KV heads to TP x EP, which
  must not exceed the 16 query heads.

Every changed script was run end to end on v6e. After this change the
scripts are sized for v6e and are not expected to fit on v5p as is.
chiajunglien added a commit to CIeNET-International/ml-auto-solutions-3 that referenced this pull request Oct 28, 2026
Move the MaxText E2E pre-training and post-training DAGs from
bodaborg-v5p-nap to bodaborg-v6e-nap. Each test uses the v6e slice size
that its MaxText script is sized for (AI-Hypercomputer/maxtext#5990).

- Add TPU_V6E_BODABORG_NAP_CLUSTER (cloud-tpu-shared-capacity,
  southamerica-west1, namespace ubench-regression-tests) and
  Region.SOUTHAMERICA_WEST1.
- Pre-training: gemma3-4b on v6e-4; gemma4-26b, qwen3-30b and
  gpt-oss-20b on v6e-32; llama3.1-70b on v6e-64.
- Post-training: gemma3-4b and qwen3-vl-2b on v6e-4 (gemma3-4b RL on
  v6e-16); qwen3-30b SFT on v6e-16; gemma4-26b, llama3.1-70b and
  gpt-oss-20b SFT on v6e-32; gpt-oss-20b RL on v6e-32; qwen3-30b,
  gemma4-26b, llama3.1-70b and qwen3.5-35b RL on v6e-64.
- gcluster: bodaborg-v6e-nap has both ct6e-standard-4t and
  ct6e-standard-8t node pools, so gcluster cannot resolve the "v6e"
  shorthand ("multiple matching machine types found in cluster for
  shorthand "v6e""). Multi-host v6e slices run only on ct6e-standard-4t
  hosts, so for v6e-16 and larger request ct6e-standard-4t with the
  slice topology. v6e-4 and v6e-8 keep the shorthand, which gcluster
  maps to a single machine type.
chiajunglien added a commit to CIeNET-International/ml-auto-solutions-3 that referenced this pull request Oct 28, 2026
Move the MaxText E2E pre-training and post-training DAGs from
bodaborg-v5p-nap to bodaborg-v6e-nap. Each test uses the v6e slice size
that its MaxText script is sized for (AI-Hypercomputer/maxtext#5990).

- Add TPU_V6E_BODABORG_NAP_CLUSTER (cloud-tpu-shared-capacity,
  southamerica-west1, namespace ubench-regression-tests) and
  Region.SOUTHAMERICA_WEST1.
- Pre-training: gemma3-4b on v6e-4; gemma4-26b, qwen3-30b and
  gpt-oss-20b on v6e-32; llama3.1-70b on v6e-64.
- Post-training: gemma3-4b and qwen3-vl-2b on v6e-4 (gemma3-4b RL on
  v6e-16); qwen3-30b SFT on v6e-16; gemma4-26b, llama3.1-70b and
  gpt-oss-20b SFT on v6e-32; gpt-oss-20b RL on v6e-32; qwen3-30b,
  gemma4-26b, llama3.1-70b and qwen3.5-35b RL on v6e-64.
- gcluster: bodaborg-v6e-nap has both ct6e-standard-4t and
  ct6e-standard-8t node pools, so gcluster cannot resolve the "v6e"
  shorthand ("multiple matching machine types found in cluster for
  shorthand "v6e""). Multi-host v6e slices run only on ct6e-standard-4t
  hosts, so for v6e-16 and larger request ct6e-standard-4t with the
  slice topology. v6e-4 and v6e-8 keep the shorthand, which gcluster
  maps to a single machine type.
chiajunglien added a commit to CIeNET-International/ml-auto-solutions-3 that referenced this pull request Oct 29, 2026
Move the MaxText E2E pre-training and post-training DAGs from
bodaborg-v5p-nap to bodaborg-v6e-nap. Each test uses the v6e slice size
that its MaxText script is sized for (AI-Hypercomputer/maxtext#5990).

- Add TPU_V6E_BODABORG_NAP_CLUSTER (cloud-tpu-shared-capacity,
  southamerica-west1, namespace ubench-regression-tests) and
  Region.SOUTHAMERICA_WEST1.
- Pre-training: gemma3-4b on v6e-4; gemma4-26b, qwen3-30b and
  gpt-oss-20b on v6e-32; llama3.1-70b on v6e-64.
- Post-training: gemma3-4b and qwen3-vl-2b on v6e-4 (gemma3-4b RL on
  v6e-16); qwen3-30b SFT on v6e-16; gemma4-26b, llama3.1-70b and
  gpt-oss-20b SFT on v6e-32; gpt-oss-20b RL on v6e-32; qwen3-30b,
  gemma4-26b, llama3.1-70b and qwen3.5-35b RL on v6e-64.
- gcluster: bodaborg-v6e-nap has both ct6e-standard-4t and
  ct6e-standard-8t node pools, so gcluster cannot resolve the "v6e"
  shorthand ("multiple matching machine types found in cluster for
  shorthand "v6e""). Multi-host v6e slices run only on ct6e-standard-4t
  hosts, so for v6e-16 and larger request ct6e-standard-4t with the
  slice topology. v6e-4 and v6e-8 keep the shorthand, which gcluster
  maps to a single machine type.
- Wait up to 3 hours, instead of the default 60 minutes, for each TPU
  training job to start. The v6e queue only has TPU quota in a daily
  4-hour window that the E2E suites share with other tests, so a job can
  queue for more than an hour. get_gke_config gets an optional
  workload_provision_timeout argument for this; its default is unchanged.
hsuan-lun-chiang pushed a commit to CIeNET-International/ml-auto-solutions-3 that referenced this pull request Nov 2, 2026
Move the MaxText E2E pre-training and post-training DAGs from
bodaborg-v5p-nap to bodaborg-v6e-nap. Each test uses the v6e slice size
that its MaxText script is sized for (AI-Hypercomputer/maxtext#5990).

- Add TPU_V6E_BODABORG_NAP_CLUSTER (cloud-tpu-shared-capacity,
  southamerica-west1, namespace ubench-regression-tests) and
  Region.SOUTHAMERICA_WEST1.
- Pre-training: gemma3-4b on v6e-4; gemma4-26b, qwen3-30b and
  gpt-oss-20b on v6e-32; llama3.1-70b on v6e-64.
- Post-training: gemma3-4b and qwen3-vl-2b on v6e-4 (gemma3-4b RL on
  v6e-16); qwen3-30b SFT on v6e-16; gemma4-26b, llama3.1-70b and
  gpt-oss-20b SFT on v6e-32; gpt-oss-20b RL on v6e-32; qwen3-30b,
  gemma4-26b, llama3.1-70b and qwen3.5-35b RL on v6e-64.
- gcluster: bodaborg-v6e-nap has both ct6e-standard-4t and
  ct6e-standard-8t node pools, so gcluster cannot resolve the "v6e"
  shorthand ("multiple matching machine types found in cluster for
  shorthand "v6e""). Multi-host v6e slices run only on ct6e-standard-4t
  hosts, so for v6e-16 and larger request ct6e-standard-4t with the
  slice topology. v6e-4 and v6e-8 keep the shorthand, which gcluster
  maps to a single machine type.
- Wait up to 3 hours, instead of the default 60 minutes, for each TPU
  training job to start. The v6e queue only has TPU quota in a daily
  4-hour window that the E2E suites share with other tests, so a job can
  queue for more than an hour. get_gke_config gets an optional
  workload_provision_timeout argument for this; its default is unchanged.
hsuan-lun-chiang pushed a commit to CIeNET-International/ml-auto-solutions-3 that referenced this pull request Nov 2, 2026
Move the MaxText E2E pre-training and post-training DAGs from
bodaborg-v5p-nap to bodaborg-v6e-nap. Each test uses the v6e slice size
that its MaxText script is sized for (AI-Hypercomputer/maxtext#5990).

- Add TPU_V6E_BODABORG_NAP_CLUSTER (cloud-tpu-shared-capacity,
  southamerica-west1, namespace ubench-regression-tests) and
  Region.SOUTHAMERICA_WEST1.
- Pre-training: gemma3-4b on v6e-4; gemma4-26b, qwen3-30b and
  gpt-oss-20b on v6e-32; llama3.1-70b on v6e-64.
- Post-training: gemma3-4b and qwen3-vl-2b on v6e-4 (gemma3-4b RL on
  v6e-16); qwen3-30b SFT on v6e-16; gemma4-26b, llama3.1-70b and
  gpt-oss-20b SFT on v6e-32; gpt-oss-20b RL on v6e-32; qwen3-30b,
  gemma4-26b, llama3.1-70b and qwen3.5-35b RL on v6e-64.
- gcluster: bodaborg-v6e-nap has both ct6e-standard-4t and
  ct6e-standard-8t node pools, so gcluster cannot resolve the "v6e"
  shorthand ("multiple matching machine types found in cluster for
  shorthand "v6e""). Multi-host v6e slices run only on ct6e-standard-4t
  hosts, so for v6e-16 and larger request ct6e-standard-4t with the
  slice topology. v6e-4 and v6e-8 keep the shorthand, which gcluster
  maps to a single machine type.
- Wait up to 3 hours, instead of the default 60 minutes, for each TPU
  training job to start. The v6e queue only has TPU quota in a daily
  4-hour window that the E2E suites share with other tests, so a job can
  queue for more than an hour. get_gke_config gets an optional
  workload_provision_timeout argument for this; its default is unchanged.
@copybara-service
copybara-service Bot merged commit 5eccd1a into main Nov 2, 2026
83 checks passed
@copybara-service
copybara-service Bot deleted the emma/e2e-scripts-v6e branch November 2, 2026 18:49
alfredyu-cienet pushed a commit to CIeNET-International/ml-auto-solutions-3 that referenced this pull request Nov 3, 2026
Move the MaxText E2E pre-training and post-training DAGs from
bodaborg-v5p-nap to bodaborg-v6e-nap. Each test uses the v6e slice size
that its MaxText script is sized for (AI-Hypercomputer/maxtext#5990).

- Add TPU_V6E_BODABORG_NAP_CLUSTER (cloud-tpu-shared-capacity,
  southamerica-west1, namespace ubench-regression-tests) and
  Region.SOUTHAMERICA_WEST1.
- Pre-training: gemma3-4b on v6e-4; gemma4-26b, qwen3-30b and
  gpt-oss-20b on v6e-32; llama3.1-70b on v6e-64.
- Post-training: gemma3-4b and qwen3-vl-2b on v6e-4 (gemma3-4b RL on
  v6e-16); qwen3-30b SFT on v6e-16; gemma4-26b, llama3.1-70b and
  gpt-oss-20b SFT on v6e-32; gpt-oss-20b RL on v6e-32; qwen3-30b,
  gemma4-26b, llama3.1-70b and qwen3.5-35b RL on v6e-64.
- gcluster: bodaborg-v6e-nap has both ct6e-standard-4t and
  ct6e-standard-8t node pools, so gcluster cannot resolve the "v6e"
  shorthand ("multiple matching machine types found in cluster for
  shorthand "v6e""). Multi-host v6e slices run only on ct6e-standard-4t
  hosts, so for v6e-16 and larger request ct6e-standard-4t with the
  slice topology. v6e-4 and v6e-8 keep the shorthand, which gcluster
  maps to a single machine type.
- Wait up to 3 hours, instead of the default 60 minutes, for each TPU
  training job to start. The v6e queue only has TPU quota in a daily
  4-hour window that the E2E suites share with other tests, so a job can
  queue for more than an hour. get_gke_config gets an optional
  workload_provision_timeout argument for this; its default is unchanged.
chiajunglien added a commit to CIeNET-International/ml-auto-solutions-3 that referenced this pull request Nov 3, 2026
…o-HF flags

Bring the to-HF changes from release/384 (GoogleCloudPlatform#1435) into the v6e migration,
and fix the Gemma4 pre-training to-HF flags.

- Wait up to 3 hours for the to-HF conversion jobs on the
  m1-megamem-96-shared CPU cluster to start. That cluster has two
  m1-megamem-96 nodes and admits two conversion jobs at a time, while one
  E2E run submits ten conversion jobs there: on 2026-11-02 the
  llama3.1-70b to-HF job was still queued after 60 minutes, so
  wait_for_workload_start timed out. get_maxtext_cpu_end_to_end_gke_config
  gets an optional workload_provision_timeout argument. The to-HF tasks on
  the n2-highmem cluster keep the 60-minute default.
- Convert the gemma4-26b pre-training checkpoint to Hugging Face with
  scan_layers=true. Since AI-Hypercomputer/maxtext#5990 the Gemma4
  pre-training script trains from the scanned checkpoint, so the to-HF
  step failed with "Your run specifies scan_layers=False, but the
  checkpoint was saved with scan_layers=True".
chiajunglien added a commit to CIeNET-International/ml-auto-solutions-3 that referenced this pull request Nov 3, 2026
…o-HF flags

The MaxText E2E to-HF conversions for the large models run on the
m1-megamem-96-shared CPU cluster after training. That cluster has two
m1-megamem-96 nodes, and its Kueue ClusterQueue admits two conversion
jobs at a time, while one E2E run submits ten conversion jobs there.
Later jobs wait longer and longer: on 2026-11-02 the llama3.1-70b to-HF
job was still queued after 60 minutes, so wait_for_workload_start timed
out and the task failed.

- get_maxtext_cpu_end_to_end_gke_config gets an optional
  workload_provision_timeout argument. Its default is unchanged, so the
  other DAGs that use it are not affected.
- In the pre-training and post-training E2E DAGs, the to-HF tasks that
  run on the m1-megamem-96 cluster wait up to 3 hours for their job to
  start. The to-HF tasks on the n2-highmem cluster keep the 60-minute
  default, so a job stuck in Pending there still fails after an hour.
- Convert the gemma4-26b pre-training checkpoint to Hugging Face with
  scan_layers=true. Since AI-Hypercomputer/maxtext#5990 the Gemma4
  pre-training script trains from the scanned checkpoint, so the to-HF
  step failed with "Your run specifies scan_layers=False, but the
  checkpoint was saved with scan_layers=True".
hsuan-lun-chiang pushed a commit to CIeNET-International/ml-auto-solutions-3 that referenced this pull request Nov 3, 2026
…o-HF flags

Bring the to-HF changes from release/384 (GoogleCloudPlatform#1435) into the v6e migration,
and fix the Gemma4 pre-training to-HF flags.

- Wait up to 3 hours for the to-HF conversion jobs on the
  m1-megamem-96-shared CPU cluster to start. That cluster has two
  m1-megamem-96 nodes and admits two conversion jobs at a time, while one
  E2E run submits ten conversion jobs there: on 2026-11-02 the
  llama3.1-70b to-HF job was still queued after 60 minutes, so
  wait_for_workload_start timed out. get_maxtext_cpu_end_to_end_gke_config
  gets an optional workload_provision_timeout argument. The to-HF tasks on
  the n2-highmem cluster keep the 60-minute default.
- Convert the gemma4-26b pre-training checkpoint to Hugging Face with
  scan_layers=true. Since AI-Hypercomputer/maxtext#5990 the Gemma4
  pre-training script trains from the scanned checkpoint, so the to-HF
  step failed with "Your run specifies scan_layers=False, but the
  checkpoint was saved with scan_layers=True".
alfredyu-cienet pushed a commit to GoogleCloudPlatform/ml-auto-solutions that referenced this pull request Nov 3, 2026
Move the MaxText E2E TPU pre-training and post-training DAGs from `bodaborg-v5p-nap` to `bodaborg-v6e-nap`. Each test runs on the v6e slice size its MaxText script is sized for in AI-Hypercomputer/maxtext#5990. This PR also includes the to-HF changes from #1435.
alfredyu-cienet pushed a commit to CIeNET-International/ml-auto-solutions-3 that referenced this pull request Nov 11, 2026
`maxtext_moe_tpu_e2e` runs `gpt-oss-20b` (`tests/end_to_end/tpu/gpt_oss/20b/test_gpt_oss_to_mt.sh` followed by `tests/end_to_end/tpu/gpt_oss/20b/test_gpt_oss.sh`) on `GkeClusters.TPU_V5P_BODABORG_NAP_CLUSTER` with default `core_count=8` (`v5p-8`, 1 slice = 4 TPU chips = 4 JAX devices).

AI-Hypercomputer/maxtext#5990 adapted the E2E training scripts for TPU v6e (*"After this change the scripts are sized for v6e and are not expected to fit on v5p as is"*) and added `ici_expert_parallelism=8` to the `maxtext.trainers.pre_train.train` command in `tests/end_to_end/tpu/gpt_oss/20b/test_gpt_oss.sh` to size it for `v6e-32` (32 chips). While GoogleCloudPlatform#1428 migrated `gpt-oss-20b` in `maxtext_e2e_tpu_pre_training` to `v6e-32` on `bodaborg-v6e-nap`, `gpt-oss-20b` in `maxtext_moe_tpu_e2e` remained on `v5p-8` (4 devices). When `fill_unspecified_mesh_axes` attempts to resolve `ici_fsdp_parallelism=-1` across 4 devices with `ici_expert_parallelism=8` (`4 / 8 = 0.5`), both the stable and nightly tasks fail before the first training step (b/582148023):

```
+ python3 -m maxtext.trainers.pre_train.train ... model_name=gpt-oss-20b scan_layers=false use_multimodal=false ici_expert_parallelism=8
...
  File "/deps/src/maxtext/utils/maxtext_utils.py", line 2208, in create_device_mesh
    ici_parallelism = max_utils.fill_unspecified_mesh_axes(ici_parallelism, num_devices_per_slice, "ICI")
  File "/deps/src/maxtext/utils/max_utils.py", line 494, in fill_unspecified_mesh_axes
    determined_val >= 1 and determined_val.is_integer()
AssertionError: Unspecified value unable to be determined with the given      ICI parallelism values
```

As suggested by AI-Hypercomputer/maxtext#5990 (and matching GoogleCloudPlatform#1428), this PR moves `gpt-oss-20b` in `maxtext_moe_tpu_e2e` from `v5p-8` to `GkeClusters.TPU_V6E_BODABORG_NAP_CLUSTER.override(core_count=32)` (`v6e-32`) and adds `"v6e-32"` to the DAG tags.
alfredyu-cienet pushed a commit to CIeNET-International/ml-auto-solutions-3 that referenced this pull request Nov 11, 2026
`maxtext_moe_tpu_e2e` runs `gpt-oss-20b` (`tests/end_to_end/tpu/gpt_oss/20b/test_gpt_oss_to_mt.sh` followed by `tests/end_to_end/tpu/gpt_oss/20b/test_gpt_oss.sh`) on `GkeClusters.TPU_V5P_BODABORG_NAP_CLUSTER` with default `core_count=8` (`v5p-8`, 1 slice = 4 TPU chips = 4 JAX devices).

AI-Hypercomputer/maxtext#5990 adapted the E2E training scripts for TPU v6e (*"After this change the scripts are sized for v6e and are not expected to fit on v5p as is"*) and added `ici_expert_parallelism=8` to the `maxtext.trainers.pre_train.train` command in `tests/end_to_end/tpu/gpt_oss/20b/test_gpt_oss.sh` to size it for `v6e-32` (32 chips). While GoogleCloudPlatform#1428 migrated `gpt-oss-20b` in `maxtext_e2e_tpu_pre_training` to `v6e-32` on `bodaborg-v6e-nap`, `gpt-oss-20b` in `maxtext_moe_tpu_e2e` remained on `v5p-8` (4 devices). When `fill_unspecified_mesh_axes` attempts to resolve `ici_fsdp_parallelism=-1` across 4 devices with `ici_expert_parallelism=8` (`4 / 8 = 0.5`), both the stable and nightly tasks fail before the first training step (b/582148023):

```
+ python3 -m maxtext.trainers.pre_train.train ... model_name=gpt-oss-20b scan_layers=false use_multimodal=false ici_expert_parallelism=8
...
  File "/deps/src/maxtext/utils/maxtext_utils.py", line 2208, in create_device_mesh
    ici_parallelism = max_utils.fill_unspecified_mesh_axes(ici_parallelism, num_devices_per_slice, "ICI")
  File "/deps/src/maxtext/utils/max_utils.py", line 494, in fill_unspecified_mesh_axes
    determined_val >= 1 and determined_val.is_integer()
AssertionError: Unspecified value unable to be determined with the given      ICI parallelism values
```

As suggested by AI-Hypercomputer/maxtext#5990 (and matching GoogleCloudPlatform#1428), this PR moves `gpt-oss-20b` in `maxtext_moe_tpu_e2e` from `v5p-8` to `GkeClusters.TPU_V6E_BODABORG_NAP_CLUSTER.override(core_count=32)` (`v6e-32`) and adds `"v6e-32"` to the DAG tags.
alfredyu-cienet pushed a commit to GoogleCloudPlatform/ml-auto-solutions that referenced this pull request Nov 12, 2026
)

`maxtext_moe_tpu_e2e` runs `gpt-oss-20b` (`tests/end_to_end/tpu/gpt_oss/20b/test_gpt_oss_to_mt.sh` followed by `tests/end_to_end/tpu/gpt_oss/20b/test_gpt_oss.sh`) on `GkeClusters.TPU_V5P_BODABORG_NAP_CLUSTER` with default `core_count=8` (`v5p-8`, 1 slice = 4 TPU chips = 4 JAX devices).

AI-Hypercomputer/maxtext#5990 adapted the E2E training scripts for TPU v6e (*"After this change the scripts are sized for v6e and are not expected to fit on v5p as is"*) and added `ici_expert_parallelism=8` to the `maxtext.trainers.pre_train.train` command in `tests/end_to_end/tpu/gpt_oss/20b/test_gpt_oss.sh` to size it for `v6e-32` (32 chips). While #1428 migrated `gpt-oss-20b` in `maxtext_e2e_tpu_pre_training` to `v6e-32` on `bodaborg-v6e-nap`, `gpt-oss-20b` in `maxtext_moe_tpu_e2e` remained on `v5p-8` (4 devices). When `fill_unspecified_mesh_axes` attempts to resolve `ici_fsdp_parallelism=-1` across 4 devices with `ici_expert_parallelism=8` (`4 / 8 = 0.5`), both the stable and nightly tasks fail before the first training step (b/582148023):

```
+ python3 -m maxtext.trainers.pre_train.train ... model_name=gpt-oss-20b scan_layers=false use_multimodal=false ici_expert_parallelism=8
...
  File "/deps/src/maxtext/utils/maxtext_utils.py", line 2208, in create_device_mesh
    ici_parallelism = max_utils.fill_unspecified_mesh_axes(ici_parallelism, num_devices_per_slice, "ICI")
  File "/deps/src/maxtext/utils/max_utils.py", line 494, in fill_unspecified_mesh_axes
    determined_val >= 1 and determined_val.is_integer()
AssertionError: Unspecified value unable to be determined with the given      ICI parallelism values
```

As suggested by AI-Hypercomputer/maxtext#5990 (and matching #1428), this PR moves `gpt-oss-20b` in `maxtext_moe_tpu_e2e` from `v5p-8` to `GkeClusters.TPU_V6E_BODABORG_NAP_CLUSTER.override(core_count=32)` (`v6e-32`) and adds `"v6e-32"` to the DAG tags.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants