You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Failing Test: tests/unit/quantizations_test.py and integration tests using fp8_full.
Error: ValueError: Invalid value configured for quantization fp8_full.
🪵 Error Details & Stack Trace
File"src/maxtext/layers/quantizations.py", line612, in_get_quant_configraiseValueError(f"Invalid value configured for quantization {config.quantization}.")
ValueError: Invalidvalueconfiguredforquantizationfp8_full.
💡 Root Cause Analysis & Context
Confidence: High (for this specific error)
The failure was introduced in PR #3912, which aimed to deprecate AQT quantization. While fp8_full was added to the QuantizationType enum in src/maxtext/configs/types.py, the corresponding logic in src/maxtext/layers/quantizations.py was not updated. Specifically:
_get_quant_config is missing fp8_full: When use_qwix_quantization is False (the default), configure_quantization calls _get_quant_config. This function lacks a branch for "fp8_full", causing it to raise a ValueError.
Broken Legacy Support: The removal of aqt_fp8 and aqt_fp8_full from the enum causes ValidationError in scripts that still use these tags (e.g., src/maxtext/inference/gpu/microbenchmark_llama2-70b_h100-8.sh), even though the underlying logic still exists in quantizations.py.
Note: While the "MaxText Package Tests" have been failing since late May (likely due to other causes such as HLO graph regressions or environment flakes), this regression will prevent the tests from passing even if those other issues are resolved.
🛠️ Recommended Fix
Update src/maxtext/layers/quantizations.py to handle fp8_full in the _get_quant_config function:
Additionally, consider restoring aqt_fp8 and aqt_fp8_full to the QuantizationType enum in src/maxtext/configs/types.py if backward compatibility is required during this phase of deprecation.
I have analyzed the recent test failures in the scheduled CI pipeline for the MaxText Package Tests workflow and identified three distinct failure modes. Below is the detailed breakdown:
Explanation: The post-training tests run on container images built from clean_py_env.Dockerfile (e.g. maxtext-unit-test-tpu:py312), which uses python:${PYTHON_VERSION}-slim-bullseye as its base. The default GCC/G++ version installed in Debian Bullseye is 10.2.1 (older than the required GCC >= 11).
Unlike maxtext_tpu_dependencies.Dockerfile which explicitly handles Bullseye and installs gcc-12 / g++-12 from Bookworm, clean_py_env.Dockerfile only installs build-essential (which resolves to default GCC 10). When install_post_train_extra_deps is executed to build vLLM, ensure_cpp20_compiler fails to locate GCC >= 11 in PATH.
🛠️ Recommended Fix
Update src/dependencies/dockerfiles/clean_py_env.Dockerfile (and potentially clean_py_env_gpu.Dockerfile) to either upgrade to a Bookworm-based base image or explicitly install a C++20 compatible compiler:
# Option A: Upgrade the base image to Debian Bookworm (highly recommended as default gcc is 12)
- FROM python:${PYTHON_VERSION}-slim-bullseye
+ FROM python:${PYTHON_VERSION}-slim-bookworm
# Option B: Or add explicit gcc-12 / g++-12 installation for Debian Bullseye (similar to maxtext_tpu_dependencies.Dockerfile)RUN if [ -f /etc/os-release ] && grep -q "bullseye" /etc/os-release; then \
echo "deb http://deb.debian.org/debian bookworm main" > /etc/apt/sources.list.d/bookworm.list && \
apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends -t bookworm gcc-12 g++-12 build-essential && \
rm -rf /var/lib/apt/lists/*; \
fi
Explanation: This failure is a compatibility regression on main where the TPU7x Tokamax FP8 MoE training path triggers a TGMM kernel configuration that results in subchannel_iters != 1. The current TGMM kernel implementation within the tokamax integration does not support subchannel iterations greater than 1, raising a NotImplementedError.
This issue is currently tracked under open issues and is expected to be resolved by the pending stacked PR series (specifically PR [Stacked PR 1/6] Import Tokamax GDN forward kernel #5151: [Stacked PR 1/5] Fix upstream pyink formatting and GMM v2 compatibility for Qwen3.5), which is not yet merged to main.
🛠️ Recommended Fix
Review and merge PR #5151 to restore TGMM and GMM compatibility for Qwen3/3.5 MoE training paths.
Error: google.api_core.exceptions.PreconditionFailed: 412 POST ... conditionNotMet ... location: If-Match
🪵 Stack Trace
FAILEDtests/integration/train_tests.py::TrainTests::test_moe_nanoo_fp8_sparse_matmul-google.api_core.exceptions.PreconditionFailed: 412POSThttps://storage.googleapis.com/upload/storage/v1/b/runner-maxtext-logs/o?uploadType=multipart&ifGenerationMatch=1788840682434862: {
"error": {
"code": 412,
"message": "At least one of the pre-conditions you specified did not hold.",
"errors": [
{
"message": "At least one of the pre-conditions you specified did not hold.",
"domain": "global",
"reason": "conditionNotMet",
"locationType": "header",
"location": "If-Match"
}
]
}
}
💡 Root Cause Analysis
Confidence: High (Infrastructure Flake)
Explanation: This is a known non-deterministic GCS API flake. During log/metadata upload, a generation match precondition mismatch occurs because of transient storage inconsistencies or concurrent write operations on the same cloud storage bucket key (runner-maxtext-logs), resulting in a 412 Precondition Failed error. This is not caused by any codebase or logic bug in MaxText.
I have analyzed the recent test failures in the scheduled CI pipeline on the main branch (Commit c55443590e2925b5da66f82ea7bc9a32a8c662e3) and identified three distinct failure modes:
🔍 Failure Mode 1: GCC Version Incompatibility for vLLM post-training extra dependencies
Error: RuntimeError: ERROR: A C++20 compatible compiler (GCC/G++ >= 11) is required to build vLLM for post-training inference...
🪵 Error Details & Stack Trace
Traceback (most recent call last):
File "/__w/maxtext/maxtext/.venv/bin/install_tpu_post_train_extra_deps", line 10, in <module>
sys.exit(main())
File "/__w/maxtext/maxtext/.venv/lib/python3.12/site-packages/dependencies/scripts/install_post_train_extra_deps.py", line 82, in main
ensure_cpp20_compiler()
File "/__w/maxtext/maxtext/.venv/lib/python3.12/site-packages/dependencies/scripts/install_post_train_extra_deps.py", line 72, in ensure_cpp20_compiler
raise RuntimeError(msg)
RuntimeError:
========================================================================
ERROR: A C++20 compatible compiler (GCC/G++ >= 11) is required to build
vLLM for post-training inference, but none was found in PATH.
========================================================================
💡 Root Cause Analysis & Context
Confidence: High (Confirmed Cause) Distinction: Codebase / Docker Environment Regression
Post-training tests run inside containers built from clean_py_env.Dockerfile (e.g. maxtext-unit-test-tpu:py312), which is based on python:${PYTHON_VERSION}-slim-bullseye. Debian Bullseye's default compiler is GCC 10.2.1, which lacks complete C++20 support. vLLM requires GCC/G++ >= 11 to compile. Unlike maxtext_tpu_dependencies.Dockerfile, clean_py_env.Dockerfile does not explicitly install a newer compiler like gcc-12, causing the dependency installation step to fail.
🛠️ Recommended Fix
We recommend updating src/dependencies/dockerfiles/clean_py_env.Dockerfile (and any other affected slim images) to either use Debian Bookworm as the base image (which defaults to GCC 12), or explicitly install gcc-12 and g++-12 when a Bullseye environment is detected.
Option A: Upgrade the base image to Bookworm
# src/dependencies/dockerfiles/clean_py_env.Dockerfile
- FROM python:${PYTHON_VERSION}-slim-bullseye
+ FROM python:${PYTHON_VERSION}-slim-bookworm
Confidence: High (Confirmed Cause) Distinction: Codebase Regression
The TPU7x Tokamax FP8 MoE training path triggers a TGMM (Tiled Matrix Multiplication) kernel configuration that resolves to subchannel_iters != 1. However, the current TGMM kernel implementation within the tokamax integration does not support subchannel iterations greater than 1, raising a NotImplementedError. This is a known regression on main and is expected to be resolved by the pending stacked PR #5151 ([Stacked PR 1/5] Fix upstream pyink formatting and GMM v2 compatibility for Qwen3.5), which is not yet merged.
🛠️ Recommended Fix
Review and merge stacked PR #5151 to restore TGMM / GMM v2 compatibility for Qwen3 / Qwen3.5 MoE training paths.
Error: google.api_core.exceptions.PreconditionFailed: 412 POST ... conditionNotMet ... location: If-Match
🪵 Error Details & Stack Trace
FAILEDtests/integration/train_tests.py::TrainTests::test_moe_nanoo_fp8_sparse_matmul-google.api_core.exceptions.PreconditionFailed: 412POSThttps://storage.googleapis.com/upload/storage/v1/b/runner-maxtext-logs/o?uploadType=multipart&ifGenerationMatch=...: {\n"error": {\n"code": 412,\n"message": "At least one of the pre-conditions you specified did not hold.",\n"errors": [\n {\n"message": "At least one of the pre-conditions you specified did not hold.",\n"domain": "global",\n"reason": "conditionNotMet",\n"locationType": "header",\n"location": "If-Match"\n }\n ]\n }\n}
💡 Root Cause Analysis & Context
Confidence: High Distinction: Infrastructure / Environment Flake
This failure is a known non-deterministic GCS API flake. During log or metadata upload, a generation match precondition mismatch occurs due to concurrent write operations or transient storage inconsistencies on the same cloud storage bucket key (runner-maxtext-logs). It is not caused by any codebase or logical bug in MaxText.
Error: google.api_core.exceptions.PreconditionFailed: 412 POST https://storage.googleapis.com/upload/storage/v1/b/runner-maxtext-logs/o?uploadType=multipart&ifGenerationMatch=1788955988933259: {"error": {"code": 412, "message": "At least one of the pre-conditions you specified did not hold."}}
🪵 Error Details & Stack Trace
self=<Blob: runner-maxtext-logs, runner_test/tensorboard/runner_test/events.out.tfevents.1788955975.linux-x86-n2-32-59h4x-runner-tcvsg-workflow, 1788955988933259>
...
google.api_core.exceptions.PreconditionFailed: 412POSThttps://storage.googleapis.com/upload/storage/v1/b/runner-maxtext-logs/o?uploadType=multipart&ifGenerationMatch=1788955988933259: {
"error": {
"code": 412,
"message": "At least one of the pre-conditions you specified did not hold.",
"errors": [
{
"message": "At least one of the pre-conditions you specified did not hold.",
"domain": "global",
"reason": "conditionNotMet",
"locationType": "header",
"location": "If-Match"
}
]
}
}
Duringhandlingoftheaboveexception, anotherexceptionoccurred:
self=<tests.integration.train_tests.TrainTeststestMethod=test_moe_fp8_sparse_matmul>
@pytest.mark.integration_testdeftest_moe_fp8_sparse_matmul(self):
>train_main(TrainTests.CONFIGS["moe_sparse"] + ["quantization=fp8"])
tests/integration/train_tests.py:541:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/maxtext/trainers/pre_train/train.py:1234: intrain_loopmetric_logger_instance.flush_metrics_and_cleanup()
src/maxtext/common/metric_logger.py:510: inflush_metrics_and_cleanupmax_utils.close_summary_writer(self.writer)
src/maxtext/utils/max_utils.py:225: inclose_summary_writersummary_writer.close()
.venv/lib/python3.12/site-packages/tensorboardX/writer.py:1251: inclosewriter.flush()
.venv/lib/python3.12/site-packages/google/cloud/storage/blob.py:3371: inupload_from_stringself.upload_from_file(
.venv/lib/python3.12/site-packages/google/cloud/storage/blob.py:3037: inupload_from_fileself._prep_and_do_upload(
_raise_from_invalid_response(exc)
Egoogle.cloud.storage.exceptions.InvalidResponse: ('Request failed with status code', 412, 'Expected one of', <HTTPStatus.OK: 200>)
💡 Root Cause Analysis & Context
Confidence: high (confirmed cause)
This failure is a classic infrastructure/environment flake rather than a codebase regression.
When running standard (non-decoupled) integration tests in parallel via pytest-xdist, multiple worker processes run on the same virtual machine and attempt to write TensorBoard logs concurrently to the same shared GCS directory under gs://runner-maxtext-logs/runner_test/tensorboard/runner_test/.
Because they share the exact same run_name=runner_test parameter from TrainTests.CONFIGS["base"], the workers generate overlapping event file uploads (e.g., matching timestamps and hostnames). When multiple processes concurrently modify/create the same GCS files using optimistic concurrency (ifGenerationMatch), GCS rightly rejects concurrent overwrites with a 412 Precondition Failed error.
🛠️ Recommended Fix
To resolve this GCS write collision and avoid parallel test flakes, we should isolate the output directories or run names for each integration test so that workers do not write to the same GCS prefix.
One straightforward fix is to override the run_name parameter dynamically with the name of the test, or assign a unique run_name in each integration test method. For example:
# In tests/integration/train_tests.py# Modify the test methods to supply a unique run_name, e.g.:train_main(TrainTests.CONFIGS["moe_sparse"] + ["quantization=fp8", "run_name=runner_test_moe_fp8_sparse_matmul"])
Error: AssertionError: Mock call assertion failed due to unexpected/missing 'item_handlers' argument
🪵 Error Details & Stack Trace
# Mismatch in Mock assertion for CheckpointManager instantiation:mock_create_mgr.assert_called_once_with(
directory=mock_config.checkpoint_dir,
options=ocp.CheckpointManagerOptions(
save_interval_steps=mock_config.checkpoint_period,
max_to_keep=mock_config.max_num_checkpoints_to_keep,
enable_async_checkpointing=mock_config.async_checkpointing,
),
)
# AssertionError: CheckpointManager was called with extra/unexpected keyword argument 'item_handlers'
💡 Root Cause Analysis & Context
Confidence: high (Confirmed Cause)
The failure was a codebase regression caused by PR #5208 ("yixuann-training-engine-support-ocdbt-zarr3").
PR #5208 modified src/maxtext/training_engine/checkpointing.py to support OCDbt and Zarr3 formats by passing item_handlers to ocp.CheckpointManager. However, the unit test test_max_text_trainer_checkpoint_manager_init in tests/post_training/unit/maxtext_engine_test.py was not updated to expect this keyword argument, causing mock assertion failures in the CI run.
🛠️ Recommended Fix
Note: This has already been resolved on main via PR #5211 ("Fix CheckpointManager init unit test failure"), which updated the unit test to patch PyTreeCheckpointHandler, assert the presence of item_handlers in CheckpointManager's mock call, and verify it is instantiated with the expected OCDbt and Zarr3 storage configurations:
I have analyzed the recent test failures in the CI pipeline and identified the following:
🔍 What Failed
Job/Matrix: MaxText Package Tests
Failing Test: N/A (Environment/Setup Phase)
Error: The test logs in .gemini/failed_logs.txt are empty (0 bytes), indicating that the failure occurred during the pipeline setup, dependency installation, or environment provisioning phase before tests were collected or executed.
🪵 Error Details & Stack Trace
# No test traceback was recorded in .gemini/failed_logs.txt.# The scheduled build failed during the setup/provisioning phase of the "MaxText Package Tests" workflow.
💡 Root Cause Analysis & Context
Confidence: moderate
Infrastructure/Environment Flake: Because the failed log file is entirely empty, the failure is highly likely related to transient infrastructure, environment configuration, or package installation issues (such as a timeout, network/GCS error, or container setup failure) rather than a codebase regression.
Codebase Status: The most recent merge on main is PR Add basic RoPE implementation to m3 core #5325 ("Add basic RoPE implementation to m3 core"). I verified that the scaffolding tests for maxtext.m3 pass successfully under system Python 3.12. The mathematical and parity layers under rope_test.py are solid and correctly mirror the legacy implementation. No syntax, import, or logical regressions were introduced by this merge.
🛠️ Recommended Fix
As this is likely an infrastructure/environment flake, we recommend triggering a re-run of the failed workflow run (#13487) to verify if the issue resolves itself. If the failure persists, please inspect the workflow's setup steps (such as package installation via uv pip or container provisioning) for transient network or dependency resolution errors.
Activity
github-actions commentedon May 26, 2026
GitHub Actions workflow MaxText Package Tests #13524 failed.
Event: schedule
Branch: main
Commit: d11185b6062096417c67b6a45e0881dfe6579222
Created by jayqi/failed-build-issue-action
github-actions commentedon May 27, 2026
GitHub Actions workflow MaxText Package Tests #13593 failed.
Event: schedule
Branch: main
Commit: cbafb8f7579a904252b250bd247641a0ceee2d22
Created by jayqi/failed-build-issue-action
github-actions commentedon May 28, 2026
GitHub Actions workflow MaxText Package Tests #13699 failed.
Event: schedule
Branch: main
Commit: fb2a3f679604388d8ca4e9490bfab9f2d0d7c304
Created by jayqi/failed-build-issue-action
github-actions commentedon May 28, 2026
GitHub Actions workflow MaxText Package Tests #13798 failed.
Event: schedule
Branch: main
Commit: f80650dfa844210f72867ebd15da5650d84ae319
Created by jayqi/failed-build-issue-action
github-actions commentedon May 29, 2026
GitHub Actions workflow MaxText Package Tests #13826 failed.
Event: schedule
Branch: main
Commit: be4fd71c8f114bfcbba98572b21cd53f06fa19c2
Created by jayqi/failed-build-issue-action
github-actions commentedon May 30, 2026
GitHub Actions workflow MaxText Package Tests #13903 failed.
Event: schedule
Branch: main
Commit: 8bc2012e8db4efd0f6683505755e65e03559896f
Created by jayqi/failed-build-issue-action
github-actions commentedon May 30, 2026
GitHub Actions workflow MaxText Package Tests #13906 failed.
Event: schedule
Branch: main
Commit: 8bc2012e8db4efd0f6683505755e65e03559896f
Created by jayqi/failed-build-issue-action
github-actions commentedon May 31, 2026
GitHub Actions workflow MaxText Package Tests #13911 failed.
Event: schedule
Branch: main
Commit: 493fba68b3fd62c6159ba39634ca1e60465ca3f1
Created by jayqi/failed-build-issue-action
github-actions commentedon Jun 1, 2026
GitHub Actions workflow MaxText Package Tests #13949 failed.
Event: schedule
Branch: main
Commit: 73293704ef336eea91c780ff73f4864213de5674
Created by jayqi/failed-build-issue-action
github-actions commentedon Jun 1, 2026
🤖 CI Failure Investigation Report
I have analyzed the recent test failures in the CI pipeline and identified a regression likely introduced in PR #3912 ("deprecate-aqt-phase1"):
🔍 What Failed
MaxText Package Tests(Scheduled/Main)tests/unit/quantizations_test.pyand integration tests usingfp8_full.ValueError: Invalid value configured for quantization fp8_full.🪵 Error Details & Stack Trace
💡 Root Cause Analysis & Context
Confidence: High (for this specific error)
The failure was introduced in PR #3912, which aimed to deprecate AQT quantization. While
fp8_fullwas added to theQuantizationTypeenum insrc/maxtext/configs/types.py, the corresponding logic insrc/maxtext/layers/quantizations.pywas not updated. Specifically:_get_quant_configis missingfp8_full: Whenuse_qwix_quantizationisFalse(the default),configure_quantizationcalls_get_quant_config. This function lacks a branch for"fp8_full", causing it to raise aValueError.aqt_fp8andaqt_fp8_fullfrom the enum causesValidationErrorin scripts that still use these tags (e.g.,src/maxtext/inference/gpu/microbenchmark_llama2-70b_h100-8.sh), even though the underlying logic still exists inquantizations.py.Note: While the "MaxText Package Tests" have been failing since late May (likely due to other causes such as HLO graph regressions or environment flakes), this regression will prevent the tests from passing even if those other issues are resolved.
🛠️ Recommended Fix
Update
src/maxtext/layers/quantizations.pyto handlefp8_fullin the_get_quant_configfunction:Additionally, consider restoring
aqt_fp8andaqt_fp8_fullto theQuantizationTypeenum insrc/maxtext/configs/types.pyif backward compatibility is required during this phase of deprecation.github-actions commentedon Jun 1, 2026
GitHub Actions workflow MaxText Package Tests #13992 failed.
Event: schedule
Branch: main
Commit: 0747df9c771f564fc3f319ad788eb2e67882594c
Created by jayqi/failed-build-issue-action
github-actions commentedon Jun 2, 2026
GitHub Actions workflow MaxText Package Tests #14020 failed.
Event: schedule
Branch: main
Commit: 575b31067a938c21af741057a184e8e78ceb36be
Created by jayqi/failed-build-issue-action
github-actions commentedon Jun 3, 2026
GitHub Actions workflow MaxText Package Tests #14128 failed.
Event: schedule
Branch: main
Commit: d8763efa5bbc2d24b15fc5c856f885698f89e01e
Created by jayqi/failed-build-issue-action
github-actions commentedon Jun 3, 2026
GitHub Actions workflow MaxText Package Tests #14135 failed.
Event: schedule
Branch: main
Commit: b74794138a28073538fdc390315c34de4a0e5a74
Created by jayqi/failed-build-issue-action
550 remaining items
github-actions commentedon Sep 8, 2026
🤖 CI Failure Investigation Report
I have analyzed the recent test failures in the scheduled CI pipeline for the MaxText Package Tests workflow and identified three distinct failure modes. Below is the detailed breakdown:
1. vLLM Post-Training Extra Dependencies Installation Failure (Codebase / Docker Environment Regression)
CPU Posttrain Tests (cpu-post-training-unit) / Execute Tests,TPU Posttrain Tests (tpu-post-training-unit) / Execute Tests,TPU7X Tests (tpu7x-post-training-unit) / Execute Tests, etc.src/dependencies/scripts/install_post_train_extra_deps.py(duringInstall the maxtext wheelstep)RuntimeError: ERROR: A C++20 compatible compiler (GCC/G++ >= 11) is required to build vLLM for post-training inference...🪵 Stack Trace
💡 Root Cause Analysis
clean_py_env.Dockerfile(e.g.maxtext-unit-test-tpu:py312), which usespython:${PYTHON_VERSION}-slim-bullseyeas its base. The default GCC/G++ version installed in Debian Bullseye is 10.2.1 (older than the required GCC >= 11).maxtext_tpu_dependencies.Dockerfilewhich explicitly handles Bullseye and installsgcc-12/g++-12from Bookworm,clean_py_env.Dockerfileonly installsbuild-essential(which resolves to default GCC 10). Wheninstall_post_train_extra_depsis executed to build vLLM,ensure_cpp20_compilerfails to locate GCC >= 11 in PATH.🛠️ Recommended Fix
Update
src/dependencies/dockerfiles/clean_py_env.Dockerfile(and potentiallyclean_py_env_gpu.Dockerfile) to either upgrade to a Bookworm-based base image or explicitly install a C++20 compatible compiler:2. TGMM subchannel_iters != 1 Incompatibility (Codebase Regression)
TPU7X Tests (tpu7x-integration) / Execute Tests (3)tests/integration/tokamax_test.py::Train::test_smoke_train_tokamax_v1_fp8_full_ep1NotImplementedError: subchannel_iters != 1 not supported yet in tgmm.🪵 Stack Trace
💡 Root Cause Analysis
mainwhere the TPU7x Tokamax FP8 MoE training path triggers a TGMM kernel configuration that results insubchannel_iters != 1. The current TGMM kernel implementation within the tokamax integration does not support subchannel iterations greater than 1, raising aNotImplementedError.[Stacked PR 1/5] Fix upstream pyink formatting and GMM v2 compatibility for Qwen3.5), which is not yet merged tomain.🛠️ Recommended Fix
Review and merge PR #5151 to restore TGMM and GMM compatibility for Qwen3/3.5 MoE training paths.
3. GCS Precondition Verification Mismatch (Infrastructure / Environment Flake)
CPU Pretrain Tests (cpu-integration) / Execute Tests (3)tests/integration/train_tests.py::TrainTests::test_moe_nanoo_fp8_sparse_matmulgoogle.api_core.exceptions.PreconditionFailed: 412 POST ... conditionNotMet ... location: If-Match🪵 Stack Trace
💡 Root Cause Analysis
runner-maxtext-logs), resulting in a412 Precondition Failederror. This is not caused by any codebase or logic bug in MaxText.github-actions commentedon Sep 8, 2026
GitHub Actions workflow MaxText Package Tests #6827 failed.
Event: schedule
Branch: main
Commit: c55443590e2925b5da66f82ea7bc9a32a8c662e3
Created by jayqi/failed-build-issue-action
github-actions commentedon Sep 8, 2026
🤖 CI Failure Investigation Report
I have analyzed the recent test failures in the scheduled CI pipeline on the
mainbranch (Commitc55443590e2925b5da66f82ea7bc9a32a8c662e3) and identified three distinct failure modes:🔍 Failure Mode 1: GCC Version Incompatibility for vLLM post-training extra dependencies
cpu-post-training-unit,tpu-post-training-unit,tpu7x-post-training-unitInstall the maxtext wheelRuntimeError: ERROR: A C++20 compatible compiler (GCC/G++ >= 11) is required to build vLLM for post-training inference...🪵 Error Details & Stack Trace
💡 Root Cause Analysis & Context
Confidence: High (Confirmed Cause)
Distinction: Codebase / Docker Environment Regression
Post-training tests run inside containers built from
clean_py_env.Dockerfile(e.g.maxtext-unit-test-tpu:py312), which is based onpython:${PYTHON_VERSION}-slim-bullseye. Debian Bullseye's default compiler is GCC 10.2.1, which lacks complete C++20 support. vLLM requires GCC/G++ >= 11 to compile. Unlikemaxtext_tpu_dependencies.Dockerfile,clean_py_env.Dockerfiledoes not explicitly install a newer compiler likegcc-12, causing the dependency installation step to fail.🛠️ Recommended Fix
We recommend updating
src/dependencies/dockerfiles/clean_py_env.Dockerfile(and any other affected slim images) to either use Debian Bookworm as the base image (which defaults to GCC 12), or explicitly installgcc-12andg++-12when a Bullseye environment is detected.Option A: Upgrade the base image to Bookworm
# src/dependencies/dockerfiles/clean_py_env.Dockerfile - FROM python:${PYTHON_VERSION}-slim-bullseye + FROM python:${PYTHON_VERSION}-slim-bookwormOption B: Install GCC 12 on Bullseye
🔍 Failure Mode 2: TGMM subchannel_iters != 1 Incompatibility
TPU7X Tests (tpu7x-integration)tests/integration/tokamax_test.py::Train::test_smoke_train_tokamax_v1_fp8_full_ep1NotImplementedError: subchannel_iters != 1 not supported yet in tgmm.🪵 Error Details & Stack Trace
💡 Root Cause Analysis & Context
Confidence: High (Confirmed Cause)
Distinction: Codebase Regression
The TPU7x Tokamax FP8 MoE training path triggers a TGMM (Tiled Matrix Multiplication) kernel configuration that resolves to
subchannel_iters != 1. However, the current TGMM kernel implementation within the tokamax integration does not support subchannel iterations greater than 1, raising aNotImplementedError. This is a known regression onmainand is expected to be resolved by the pending stacked PR #5151 ([Stacked PR 1/5] Fix upstream pyink formatting and GMM v2 compatibility for Qwen3.5), which is not yet merged.🛠️ Recommended Fix
Review and merge stacked PR #5151 to restore TGMM / GMM v2 compatibility for Qwen3 / Qwen3.5 MoE training paths.
🔍 Failure Mode 3: GCS Precondition Verification Mismatch
CPU Pretrain Tests (cpu-integration)tests/integration/train_tests.py::TrainTests::test_moe_nanoo_fp8_sparse_matmulgoogle.api_core.exceptions.PreconditionFailed: 412 POST ... conditionNotMet ... location: If-Match🪵 Error Details & Stack Trace
💡 Root Cause Analysis & Context
Confidence: High
Distinction: Infrastructure / Environment Flake
This failure is a known non-deterministic GCS API flake. During log or metadata upload, a generation match precondition mismatch occurs due to concurrent write operations or transient storage inconsistencies on the same cloud storage bucket key (
runner-maxtext-logs). It is not caused by any codebase or logical bug in MaxText.github-actions commentedon Sep 8, 2026
GitHub Actions workflow MaxText Package Tests #6834 failed.
Event: schedule
Branch: main
Commit: c55443590e2925b5da66f82ea7bc9a32a8c662e3
Created by jayqi/failed-build-issue-action
github-actions commentedon Sep 8, 2026
GitHub Actions workflow MaxText Package Tests #6855 failed.
Event: schedule
Branch: main
Commit: 81672e7630ac05d1d88a5c9e40f7cc8a7c1575a2
Created by jayqi/failed-build-issue-action
github-actions commentedon Sep 9, 2026
GitHub Actions workflow MaxText Package Tests #6943 failed.
Event: schedule
Branch: main
Commit: c53e1af89f5b44b1608a31e8d9f5dda5dcb057ef
Created by jayqi/failed-build-issue-action
github-actions commentedon Sep 9, 2026
GitHub Actions workflow MaxText Package Tests #6947 failed.
Event: schedule
Branch: main
Commit: 4f062c5e613092e860096d9bed50d845d268ec3f
Created by jayqi/failed-build-issue-action
github-actions commentedon Sep 9, 2026
GitHub Actions workflow MaxText Package Tests #6950 failed.
Event: schedule
Branch: main
Commit: 4f062c5e613092e860096d9bed50d845d268ec3f
Created by jayqi/failed-build-issue-action
github-actions commentedon Sep 9, 2026
🤖 CI Failure Investigation Report
I have analyzed the recent test failures in the CI pipeline and identified the following:
🔍 What Failed
cpu-integrationtests/integration/train_tests.py::TrainTests::test_moe_fp8_sparse_matmulgoogle.api_core.exceptions.PreconditionFailed: 412 POST https://storage.googleapis.com/upload/storage/v1/b/runner-maxtext-logs/o?uploadType=multipart&ifGenerationMatch=1788955988933259: {"error": {"code": 412, "message": "At least one of the pre-conditions you specified did not hold."}}🪵 Error Details & Stack Trace
💡 Root Cause Analysis & Context
Confidence: high (confirmed cause)
This failure is a classic infrastructure/environment flake rather than a codebase regression.
When running standard (non-decoupled) integration tests in parallel via
pytest-xdist, multiple worker processes run on the same virtual machine and attempt to write TensorBoard logs concurrently to the same shared GCS directory undergs://runner-maxtext-logs/runner_test/tensorboard/runner_test/.Because they share the exact same
run_name=runner_testparameter fromTrainTests.CONFIGS["base"], the workers generate overlapping event file uploads (e.g., matching timestamps and hostnames). When multiple processes concurrently modify/create the same GCS files using optimistic concurrency (ifGenerationMatch), GCS rightly rejects concurrent overwrites with a412 Precondition Failederror.🛠️ Recommended Fix
To resolve this GCS write collision and avoid parallel test flakes, we should isolate the output directories or run names for each integration test so that workers do not write to the same GCS prefix.
One straightforward fix is to override the
run_nameparameter dynamically with the name of the test, or assign a uniquerun_namein each integration test method. For example:github-actions commentedon Sep 10, 2026
GitHub Actions workflow MaxText Package Tests #7135 failed.
Event: schedule
Branch: main
Commit: 1172f2f0bcf9ea4da914d4d7c54e0622e386e917
Created by jayqi/failed-build-issue-action
github-actions commentedon Sep 10, 2026
GitHub Actions workflow MaxText Package Tests #7143 failed.
Event: schedule
Branch: main
Commit: 7709f944888968e3d81d83a5d096310dbdc2b424
Created by jayqi/failed-build-issue-action
github-actions commentedon Sep 12, 2026
🤖 CI Failure Investigation Report
I have analyzed the recent test failures in the CI pipeline and identified the following:
🔍 What Failed
tests/post_training/unit/maxtext_engine_test.py::MaxTextTrainingEngineTest::test_max_text_trainer_checkpoint_manager_initAssertionError: Mock call assertion failed due to unexpected/missing 'item_handlers' argument🪵 Error Details & Stack Trace
💡 Root Cause Analysis & Context
Confidence: high (Confirmed Cause)
The failure was a codebase regression caused by PR #5208 ("yixuann-training-engine-support-ocdbt-zarr3").
PR #5208 modified
src/maxtext/training_engine/checkpointing.pyto support OCDbt and Zarr3 formats by passingitem_handlerstoocp.CheckpointManager. However, the unit testtest_max_text_trainer_checkpoint_manager_initintests/post_training/unit/maxtext_engine_test.pywas not updated to expect this keyword argument, causing mock assertion failures in the CI run.🛠️ Recommended Fix
Note: This has already been resolved on
mainvia PR #5211 ("Fix CheckpointManager init unit test failure"), which updated the unit test to patchPyTreeCheckpointHandler, assert the presence ofitem_handlersinCheckpointManager's mock call, and verify it is instantiated with the expected OCDbt and Zarr3 storage configurations:github-actions commentedon Sep 22, 2026
🤖 CI Failure Investigation Report
I have analyzed the recent test failures in the CI pipeline and identified the following:
🔍 What Failed
MaxText Package Tests.gemini/failed_logs.txtare empty (0 bytes), indicating that the failure occurred during the pipeline setup, dependency installation, or environment provisioning phase before tests were collected or executed.🪵 Error Details & Stack Trace
💡 Root Cause Analysis & Context
Confidence: moderate
mainis PR Add basic RoPE implementation to m3 core #5325 ("Add basic RoPE implementation to m3 core"). I verified that the scaffolding tests formaxtext.m3pass successfully under system Python 3.12. The mathematical and parity layers underrope_test.pyare solid and correctly mirror the legacy implementation. No syntax, import, or logical regressions were introduced by this merge.🛠️ Recommended Fix
As this is likely an infrastructure/environment flake, we recommend triggering a re-run of the failed workflow run (#13487) to verify if the issue resolves itself. If the failure persists, please inspect the workflow's setup steps (such as package installation via
uv pipor container provisioning) for transient network or dependency resolution errors.