Skip to content

Failed build: MaxText Package Tests #3979

Closed

Description

@github-actions

GitHub Actions workflow MaxText Package Tests #13487 failed.

Event: schedule
Branch: main
Commit: 4110d139cc2a8a3d4fab53cd5568028a7897568f

Created by jayqi/failed-build-issue-action

Activity

  1. github-actions commented on May 26, 2026

    @github-actions
    ContributorAuthor

    GitHub Actions workflow MaxText Package Tests #13524 failed.

    Event: schedule
    Branch: main
    Commit: d11185b6062096417c67b6a45e0881dfe6579222

    Created by jayqi/failed-build-issue-action

  2. github-actions commented on May 27, 2026

    @github-actions
    ContributorAuthor

    GitHub Actions workflow MaxText Package Tests #13593 failed.

    Event: schedule
    Branch: main
    Commit: cbafb8f7579a904252b250bd247641a0ceee2d22

    Created by jayqi/failed-build-issue-action

  3. github-actions commented on May 28, 2026

    @github-actions
    ContributorAuthor

    GitHub Actions workflow MaxText Package Tests #13699 failed.

    Event: schedule
    Branch: main
    Commit: fb2a3f679604388d8ca4e9490bfab9f2d0d7c304

    Created by jayqi/failed-build-issue-action

  4. github-actions commented on May 28, 2026

    @github-actions
    ContributorAuthor

    GitHub Actions workflow MaxText Package Tests #13798 failed.

    Event: schedule
    Branch: main
    Commit: f80650dfa844210f72867ebd15da5650d84ae319

    Created by jayqi/failed-build-issue-action

  5. github-actions commented on May 29, 2026

    @github-actions
    ContributorAuthor

    GitHub Actions workflow MaxText Package Tests #13826 failed.

    Event: schedule
    Branch: main
    Commit: be4fd71c8f114bfcbba98572b21cd53f06fa19c2

    Created by jayqi/failed-build-issue-action

  6. github-actions commented on May 30, 2026

    @github-actions
    ContributorAuthor

    GitHub Actions workflow MaxText Package Tests #13903 failed.

    Event: schedule
    Branch: main
    Commit: 8bc2012e8db4efd0f6683505755e65e03559896f

    Created by jayqi/failed-build-issue-action

  7. github-actions commented on May 30, 2026

    @github-actions
    ContributorAuthor

    GitHub Actions workflow MaxText Package Tests #13906 failed.

    Event: schedule
    Branch: main
    Commit: 8bc2012e8db4efd0f6683505755e65e03559896f

    Created by jayqi/failed-build-issue-action

  8. github-actions commented on May 31, 2026

    @github-actions
    ContributorAuthor

    GitHub Actions workflow MaxText Package Tests #13911 failed.

    Event: schedule
    Branch: main
    Commit: 493fba68b3fd62c6159ba39634ca1e60465ca3f1

    Created by jayqi/failed-build-issue-action

  9. github-actions commented on Jun 1, 2026

    @github-actions
    ContributorAuthor

    GitHub Actions workflow MaxText Package Tests #13949 failed.

    Event: schedule
    Branch: main
    Commit: 73293704ef336eea91c780ff73f4864213de5674

    Created by jayqi/failed-build-issue-action

  10. github-actions commented on Jun 1, 2026

    @github-actions
    ContributorAuthor

    🤖 CI Failure Investigation Report

    I have analyzed the recent test failures in the CI pipeline and identified a regression likely introduced in PR #3912 ("deprecate-aqt-phase1"):

    🔍 What Failed

    • Job/Matrix: MaxText Package Tests (Scheduled/Main)
    • Failing Test: tests/unit/quantizations_test.py and integration tests using fp8_full.
    • Error: ValueError: Invalid value configured for quantization fp8_full.

    🪵 Error Details & Stack Trace

      File "src/maxtext/layers/quantizations.py", line 612, in _get_quant_config
        raise ValueError(f"Invalid value configured for quantization {config.quantization}.")
    ValueError: Invalid value configured for quantization fp8_full.

    💡 Root Cause Analysis & Context

    Confidence: High (for this specific error)

    The failure was introduced in PR #3912, which aimed to deprecate AQT quantization. While fp8_full was added to the QuantizationType enum in src/maxtext/configs/types.py, the corresponding logic in src/maxtext/layers/quantizations.py was not updated. Specifically:

    1. _get_quant_config is missing fp8_full: When use_qwix_quantization is False (the default), configure_quantization calls _get_quant_config. This function lacks a branch for "fp8_full", causing it to raise a ValueError.
    2. Broken Legacy Support: The removal of aqt_fp8 and aqt_fp8_full from the enum causes ValidationError in scripts that still use these tags (e.g., src/maxtext/inference/gpu/microbenchmark_llama2-70b_h100-8.sh), even though the underlying logic still exists in quantizations.py.

    Note: While the "MaxText Package Tests" have been failing since late May (likely due to other causes such as HLO graph regressions or environment flakes), this regression will prevent the tests from passing even if those other issues are resolved.

    🛠️ Recommended Fix

    Update src/maxtext/layers/quantizations.py to handle fp8_full in the _get_quant_config function:

    <<<<
      if config.quantization == "aqt_fp8_full":
        return _get_aqt_fp8_default_config(config)
    ====
      if config.quantization == "aqt_fp8_full" or config.quantization == "fp8_full":
        return _get_aqt_fp8_default_config(config)
    >>>>

    Additionally, consider restoring aqt_fp8 and aqt_fp8_full to the QuantizationType enum in src/maxtext/configs/types.py if backward compatibility is required during this phase of deprecation.

  11. github-actions commented on Jun 1, 2026

    @github-actions
    ContributorAuthor

    GitHub Actions workflow MaxText Package Tests #13992 failed.

    Event: schedule
    Branch: main
    Commit: 0747df9c771f564fc3f319ad788eb2e67882594c

    Created by jayqi/failed-build-issue-action

  12. github-actions commented on Jun 2, 2026

    @github-actions
    ContributorAuthor

    GitHub Actions workflow MaxText Package Tests #14020 failed.

    Event: schedule
    Branch: main
    Commit: 575b31067a938c21af741057a184e8e78ceb36be

    Created by jayqi/failed-build-issue-action

  13. github-actions commented on Jun 3, 2026

    @github-actions
    ContributorAuthor

    GitHub Actions workflow MaxText Package Tests #14128 failed.

    Event: schedule
    Branch: main
    Commit: d8763efa5bbc2d24b15fc5c856f885698f89e01e

    Created by jayqi/failed-build-issue-action

  14. github-actions commented on Jun 3, 2026

    @github-actions
    ContributorAuthor

    GitHub Actions workflow MaxText Package Tests #14135 failed.

    Event: schedule
    Branch: main
    Commit: b74794138a28073538fdc390315c34de4a0e5a74

    Created by jayqi/failed-build-issue-action

  15. 550 remaining items

  16. github-actions commented on Sep 8, 2026

    @github-actions
    ContributorAuthor

    🤖 CI Failure Investigation Report

    I have analyzed the recent test failures in the scheduled CI pipeline for the MaxText Package Tests workflow and identified three distinct failure modes. Below is the detailed breakdown:


    1. vLLM Post-Training Extra Dependencies Installation Failure (Codebase / Docker Environment Regression)

    • Job/Matrix: CPU Posttrain Tests (cpu-post-training-unit) / Execute Tests, TPU Posttrain Tests (tpu-post-training-unit) / Execute Tests, TPU7X Tests (tpu7x-post-training-unit) / Execute Tests, etc.
    • Failing Script: src/dependencies/scripts/install_post_train_extra_deps.py (during Install the maxtext wheel step)
    • Error: RuntimeError: ERROR: A C++20 compatible compiler (GCC/G++ >= 11) is required to build vLLM for post-training inference...

    🪵 Stack Trace

    Traceback (most recent call last):
      File "/__w/maxtext/maxtext/.venv/bin/install_tpu_post_train_extra_deps", line 10, in <module>
        sys.exit(main())
      File "/__w/maxtext/maxtext/.venv/lib/python3.12/site-packages/dependencies/scripts/install_post_train_extra_deps.py", line 82, in main
        ensure_cpp20_compiler()
      File "/__w/maxtext/maxtext/.venv/lib/python3.12/site-packages/dependencies/scripts/install_post_train_extra_deps.py", line 72, in ensure_cpp20_compiler
        raise RuntimeError(msg)
    RuntimeError: 
    ========================================================================
    ERROR: A C++20 compatible compiler (GCC/G++ >= 11) is required to build
    vLLM for post-training inference, but none was found in PATH.
    ========================================================================

    💡 Root Cause Analysis

    • Confidence: Confirmed (High)
    • Explanation: The post-training tests run on container images built from clean_py_env.Dockerfile (e.g. maxtext-unit-test-tpu:py312), which uses python:${PYTHON_VERSION}-slim-bullseye as its base. The default GCC/G++ version installed in Debian Bullseye is 10.2.1 (older than the required GCC >= 11).
    • Unlike maxtext_tpu_dependencies.Dockerfile which explicitly handles Bullseye and installs gcc-12 / g++-12 from Bookworm, clean_py_env.Dockerfile only installs build-essential (which resolves to default GCC 10). When install_post_train_extra_deps is executed to build vLLM, ensure_cpp20_compiler fails to locate GCC >= 11 in PATH.

    🛠️ Recommended Fix

    Update src/dependencies/dockerfiles/clean_py_env.Dockerfile (and potentially clean_py_env_gpu.Dockerfile) to either upgrade to a Bookworm-based base image or explicitly install a C++20 compatible compiler:

    # Option A: Upgrade the base image to Debian Bookworm (highly recommended as default gcc is 12)
    - FROM python:${PYTHON_VERSION}-slim-bullseye
    + FROM python:${PYTHON_VERSION}-slim-bookworm
    
    # Option B: Or add explicit gcc-12 / g++-12 installation for Debian Bullseye (similar to maxtext_tpu_dependencies.Dockerfile)
    RUN if [ -f /etc/os-release ] && grep -q "bullseye" /etc/os-release; then \
            echo "deb http://deb.debian.org/debian bookworm main" > /etc/apt/sources.list.d/bookworm.list && \
            apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends -t bookworm gcc-12 g++-12 build-essential && \
            rm -rf /var/lib/apt/lists/*; \
        fi

    2. TGMM subchannel_iters != 1 Incompatibility (Codebase Regression)

    • Job/Matrix: TPU7X Tests (tpu7x-integration) / Execute Tests (3)
    • Failing Test: tests/integration/tokamax_test.py::Train::test_smoke_train_tokamax_v1_fp8_full_ep1
    • Error: NotImplementedError: subchannel_iters != 1 not supported yet in tgmm.

    🪵 Stack Trace

    FAILED tests/integration/tokamax_test.py::Train::test_smoke_train_tokamax_v1_fp8_full_ep1 - NotImplementedError: subchannel_iters != 1 not supported yet in tgmm.

    💡 Root Cause Analysis

    • Confidence: High
    • Explanation: This failure is a compatibility regression on main where the TPU7x Tokamax FP8 MoE training path triggers a TGMM kernel configuration that results in subchannel_iters != 1. The current TGMM kernel implementation within the tokamax integration does not support subchannel iterations greater than 1, raising a NotImplementedError.
    • This issue is currently tracked under open issues and is expected to be resolved by the pending stacked PR series (specifically PR [Stacked PR 1/6] Import Tokamax GDN forward kernel #5151: [Stacked PR 1/5] Fix upstream pyink formatting and GMM v2 compatibility for Qwen3.5), which is not yet merged to main.

    🛠️ Recommended Fix

    Review and merge PR #5151 to restore TGMM and GMM compatibility for Qwen3/3.5 MoE training paths.


    3. GCS Precondition Verification Mismatch (Infrastructure / Environment Flake)

    • Job/Matrix: CPU Pretrain Tests (cpu-integration) / Execute Tests (3)
    • Failing Test: tests/integration/train_tests.py::TrainTests::test_moe_nanoo_fp8_sparse_matmul
    • Error: google.api_core.exceptions.PreconditionFailed: 412 POST ... conditionNotMet ... location: If-Match

    🪵 Stack Trace

    FAILED tests/integration/train_tests.py::TrainTests::test_moe_nanoo_fp8_sparse_matmul - google.api_core.exceptions.PreconditionFailed: 412 POST https://storage.googleapis.com/upload/storage/v1/b/runner-maxtext-logs/o?uploadType=multipart&ifGenerationMatch=1788840682434862: {
      "error": {
        "code": 412,
        "message": "At least one of the pre-conditions you specified did not hold.",
        "errors": [
          {
            "message": "At least one of the pre-conditions you specified did not hold.",
            "domain": "global",
            "reason": "conditionNotMet",
            "locationType": "header",
            "location": "If-Match"
          }
        ]
      }
    }

    💡 Root Cause Analysis

    • Confidence: High (Infrastructure Flake)
    • Explanation: This is a known non-deterministic GCS API flake. During log/metadata upload, a generation match precondition mismatch occurs because of transient storage inconsistencies or concurrent write operations on the same cloud storage bucket key (runner-maxtext-logs), resulting in a 412 Precondition Failed error. This is not caused by any codebase or logic bug in MaxText.
  17. github-actions commented on Sep 8, 2026

    @github-actions
    ContributorAuthor

    GitHub Actions workflow MaxText Package Tests #6827 failed.

    Event: schedule
    Branch: main
    Commit: c55443590e2925b5da66f82ea7bc9a32a8c662e3

    Created by jayqi/failed-build-issue-action

  18. github-actions commented on Sep 8, 2026

    @github-actions
    ContributorAuthor

    🤖 CI Failure Investigation Report

    I have analyzed the recent test failures in the scheduled CI pipeline on the main branch (Commit c55443590e2925b5da66f82ea7bc9a32a8c662e3) and identified three distinct failure modes:


    🔍 Failure Mode 1: GCC Version Incompatibility for vLLM post-training extra dependencies

    • Job/Matrix: cpu-post-training-unit, tpu-post-training-unit, tpu7x-post-training-unit
    • Failing Step: Install the maxtext wheel
    • Error: RuntimeError: ERROR: A C++20 compatible compiler (GCC/G++ >= 11) is required to build vLLM for post-training inference...

    🪵 Error Details & Stack Trace

    Traceback (most recent call last):
      File "/__w/maxtext/maxtext/.venv/bin/install_tpu_post_train_extra_deps", line 10, in <module>
        sys.exit(main())
      File "/__w/maxtext/maxtext/.venv/lib/python3.12/site-packages/dependencies/scripts/install_post_train_extra_deps.py", line 82, in main
        ensure_cpp20_compiler()
      File "/__w/maxtext/maxtext/.venv/lib/python3.12/site-packages/dependencies/scripts/install_post_train_extra_deps.py", line 72, in ensure_cpp20_compiler
        raise RuntimeError(msg)
    RuntimeError: 
    ========================================================================
    ERROR: A C++20 compatible compiler (GCC/G++ >= 11) is required to build
    vLLM for post-training inference, but none was found in PATH.
    ========================================================================
    

    💡 Root Cause Analysis & Context

    Confidence: High (Confirmed Cause)
    Distinction: Codebase / Docker Environment Regression

    Post-training tests run inside containers built from clean_py_env.Dockerfile (e.g. maxtext-unit-test-tpu:py312), which is based on python:${PYTHON_VERSION}-slim-bullseye. Debian Bullseye's default compiler is GCC 10.2.1, which lacks complete C++20 support. vLLM requires GCC/G++ >= 11 to compile. Unlike maxtext_tpu_dependencies.Dockerfile, clean_py_env.Dockerfile does not explicitly install a newer compiler like gcc-12, causing the dependency installation step to fail.

    🛠️ Recommended Fix

    We recommend updating src/dependencies/dockerfiles/clean_py_env.Dockerfile (and any other affected slim images) to either use Debian Bookworm as the base image (which defaults to GCC 12), or explicitly install gcc-12 and g++-12 when a Bullseye environment is detected.

    Option A: Upgrade the base image to Bookworm

    # src/dependencies/dockerfiles/clean_py_env.Dockerfile
    - FROM python:${PYTHON_VERSION}-slim-bullseye
    + FROM python:${PYTHON_VERSION}-slim-bookworm

    Option B: Install GCC 12 on Bullseye

    # Add to clean_py_env.Dockerfile:
    RUN if [ -f /etc/os-release ] && grep -q "bullseye" /etc/os-release; then \
            echo "deb http://deb.debian.org/debian bookworm main" > /etc/apt/sources.list.d/bookworm.list && \
            apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends -t bookworm gcc-12 g++-12 build-essential && \
            rm -rf /var/lib/apt/lists/*; \
        fi

    🔍 Failure Mode 2: TGMM subchannel_iters != 1 Incompatibility

    • Job/Matrix: TPU7X Tests (tpu7x-integration)
    • Failing Test: tests/integration/tokamax_test.py::Train::test_smoke_train_tokamax_v1_fp8_full_ep1
    • Error: NotImplementedError: subchannel_iters != 1 not supported yet in tgmm.

    🪵 Error Details & Stack Trace

    FAILED tests/integration/tokamax_test.py::Train::test_smoke_train_tokamax_v1_fp8_full_ep1 - NotImplementedError: subchannel_iters != 1 not supported yet in tgmm.

    💡 Root Cause Analysis & Context

    Confidence: High (Confirmed Cause)
    Distinction: Codebase Regression

    The TPU7x Tokamax FP8 MoE training path triggers a TGMM (Tiled Matrix Multiplication) kernel configuration that resolves to subchannel_iters != 1. However, the current TGMM kernel implementation within the tokamax integration does not support subchannel iterations greater than 1, raising a NotImplementedError. This is a known regression on main and is expected to be resolved by the pending stacked PR #5151 ([Stacked PR 1/5] Fix upstream pyink formatting and GMM v2 compatibility for Qwen3.5), which is not yet merged.

    🛠️ Recommended Fix

    Review and merge stacked PR #5151 to restore TGMM / GMM v2 compatibility for Qwen3 / Qwen3.5 MoE training paths.


    🔍 Failure Mode 3: GCS Precondition Verification Mismatch

    • Job/Matrix: CPU Pretrain Tests (cpu-integration)
    • Failing Test: tests/integration/train_tests.py::TrainTests::test_moe_nanoo_fp8_sparse_matmul
    • Error: google.api_core.exceptions.PreconditionFailed: 412 POST ... conditionNotMet ... location: If-Match

    🪵 Error Details & Stack Trace

    FAILED tests/integration/train_tests.py::TrainTests::test_moe_nanoo_fp8_sparse_matmul - google.api_core.exceptions.PreconditionFailed: 412 POST https://storage.googleapis.com/upload/storage/v1/b/runner-maxtext-logs/o?uploadType=multipart&ifGenerationMatch=...: {\n  "error": {\n    "code": 412,\n    "message": "At least one of the pre-conditions you specified did not hold.",\n    "errors": [\n      {\n        "message": "At least one of the pre-conditions you specified did not hold.",\n        "domain": "global",\n        "reason": "conditionNotMet",\n        "locationType": "header",\n        "location": "If-Match"\n      }\n    ]\n  }\n}

    💡 Root Cause Analysis & Context

    Confidence: High
    Distinction: Infrastructure / Environment Flake

    This failure is a known non-deterministic GCS API flake. During log or metadata upload, a generation match precondition mismatch occurs due to concurrent write operations or transient storage inconsistencies on the same cloud storage bucket key (runner-maxtext-logs). It is not caused by any codebase or logical bug in MaxText.


  19. github-actions commented on Sep 8, 2026

    @github-actions
    ContributorAuthor

    GitHub Actions workflow MaxText Package Tests #6834 failed.

    Event: schedule
    Branch: main
    Commit: c55443590e2925b5da66f82ea7bc9a32a8c662e3

    Created by jayqi/failed-build-issue-action

  20. github-actions commented on Sep 8, 2026

    @github-actions
    ContributorAuthor

    GitHub Actions workflow MaxText Package Tests #6855 failed.

    Event: schedule
    Branch: main
    Commit: 81672e7630ac05d1d88a5c9e40f7cc8a7c1575a2

    Created by jayqi/failed-build-issue-action

  21. github-actions commented on Sep 9, 2026

    @github-actions
    ContributorAuthor

    GitHub Actions workflow MaxText Package Tests #6943 failed.

    Event: schedule
    Branch: main
    Commit: c53e1af89f5b44b1608a31e8d9f5dda5dcb057ef

    Created by jayqi/failed-build-issue-action

  22. github-actions commented on Sep 9, 2026

    @github-actions
    ContributorAuthor

    GitHub Actions workflow MaxText Package Tests #6947 failed.

    Event: schedule
    Branch: main
    Commit: 4f062c5e613092e860096d9bed50d845d268ec3f

    Created by jayqi/failed-build-issue-action

  23. github-actions commented on Sep 9, 2026

    @github-actions
    ContributorAuthor

    GitHub Actions workflow MaxText Package Tests #6950 failed.

    Event: schedule
    Branch: main
    Commit: 4f062c5e613092e860096d9bed50d845d268ec3f

    Created by jayqi/failed-build-issue-action

  24. github-actions commented on Sep 9, 2026

    @github-actions
    ContributorAuthor

    🤖 CI Failure Investigation Report

    I have analyzed the recent test failures in the CI pipeline and identified the following:

    🔍 What Failed

    • Job/Matrix: cpu-integration
    • Failing Test: tests/integration/train_tests.py::TrainTests::test_moe_fp8_sparse_matmul
    • Error: google.api_core.exceptions.PreconditionFailed: 412 POST https://storage.googleapis.com/upload/storage/v1/b/runner-maxtext-logs/o?uploadType=multipart&ifGenerationMatch=1788955988933259: {"error": {"code": 412, "message": "At least one of the pre-conditions you specified did not hold."}}

    🪵 Error Details & Stack Trace

    self = <Blob: runner-maxtext-logs, runner_test/tensorboard/runner_test/events.out.tfevents.1788955975.linux-x86-n2-32-59h4x-runner-tcvsg-workflow, 1788955988933259>
    ...
    google.api_core.exceptions.PreconditionFailed: 412 POST https://storage.googleapis.com/upload/storage/v1/b/runner-maxtext-logs/o?uploadType=multipart&ifGenerationMatch=1788955988933259: {
      "error": {
        "code": 412,
        "message": "At least one of the pre-conditions you specified did not hold.",
        "errors": [
          {
            "message": "At least one of the pre-conditions you specified did not hold.",
            "domain": "global",
            "reason": "conditionNotMet",
            "locationType": "header",
            "location": "If-Match"
          }
        ]
      }
    }
    During handling of the above exception, another exception occurred:
    
    self = <tests.integration.train_tests.TrainTests testMethod=test_moe_fp8_sparse_matmul>
    
        @pytest.mark.integration_test
        def test_moe_fp8_sparse_matmul(self):
    >     train_main(TrainTests.CONFIGS["moe_sparse"] + ["quantization=fp8"])
    
    tests/integration/train_tests.py:541: 
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    src/maxtext/trainers/pre_train/train.py:1234: in train_loop
        metric_logger_instance.flush_metrics_and_cleanup()
    src/maxtext/common/metric_logger.py:510: in flush_metrics_and_cleanup
        max_utils.close_summary_writer(self.writer)
    src/maxtext/utils/max_utils.py:225: in close_summary_writer
        summary_writer.close()
    .venv/lib/python3.12/site-packages/tensorboardX/writer.py:1251: in close
        writer.flush()
    .venv/lib/python3.12/site-packages/google/cloud/storage/blob.py:3371: in upload_from_string
        self.upload_from_file(
    .venv/lib/python3.12/site-packages/google/cloud/storage/blob.py:3037: in upload_from_file
        self._prep_and_do_upload(
    _raise_from_invalid_response(exc)
    E           google.cloud.storage.exceptions.InvalidResponse: ('Request failed with status code', 412, 'Expected one of', <HTTPStatus.OK: 200>)

    💡 Root Cause Analysis & Context

    Confidence: high (confirmed cause)

    This failure is a classic infrastructure/environment flake rather than a codebase regression.

    When running standard (non-decoupled) integration tests in parallel via pytest-xdist, multiple worker processes run on the same virtual machine and attempt to write TensorBoard logs concurrently to the same shared GCS directory under gs://runner-maxtext-logs/runner_test/tensorboard/runner_test/.

    Because they share the exact same run_name=runner_test parameter from TrainTests.CONFIGS["base"], the workers generate overlapping event file uploads (e.g., matching timestamps and hostnames). When multiple processes concurrently modify/create the same GCS files using optimistic concurrency (ifGenerationMatch), GCS rightly rejects concurrent overwrites with a 412 Precondition Failed error.

    🛠️ Recommended Fix

    To resolve this GCS write collision and avoid parallel test flakes, we should isolate the output directories or run names for each integration test so that workers do not write to the same GCS prefix.

    One straightforward fix is to override the run_name parameter dynamically with the name of the test, or assign a unique run_name in each integration test method. For example:

    # In tests/integration/train_tests.py
    # Modify the test methods to supply a unique run_name, e.g.:
    train_main(TrainTests.CONFIGS["moe_sparse"] + ["quantization=fp8", "run_name=runner_test_moe_fp8_sparse_matmul"])
  25. github-actions commented on Sep 10, 2026

    @github-actions
    ContributorAuthor

    GitHub Actions workflow MaxText Package Tests #7135 failed.

    Event: schedule
    Branch: main
    Commit: 1172f2f0bcf9ea4da914d4d7c54e0622e386e917

    Created by jayqi/failed-build-issue-action

  26. github-actions commented on Sep 10, 2026

    @github-actions
    ContributorAuthor

    GitHub Actions workflow MaxText Package Tests #7143 failed.

    Event: schedule
    Branch: main
    Commit: 7709f944888968e3d81d83a5d096310dbdc2b424

    Created by jayqi/failed-build-issue-action

  27. github-actions commented on Sep 12, 2026

    @github-actions
    ContributorAuthor

    🤖 CI Failure Investigation Report

    I have analyzed the recent test failures in the CI pipeline and identified the following:

    🔍 What Failed

    • Job/Matrix: MaxText Package Tests
    • Failing Test: tests/post_training/unit/maxtext_engine_test.py::MaxTextTrainingEngineTest::test_max_text_trainer_checkpoint_manager_init
    • Error: AssertionError: Mock call assertion failed due to unexpected/missing 'item_handlers' argument

    🪵 Error Details & Stack Trace

    # Mismatch in Mock assertion for CheckpointManager instantiation:
    mock_create_mgr.assert_called_once_with(
        directory=mock_config.checkpoint_dir,
        options=ocp.CheckpointManagerOptions(
            save_interval_steps=mock_config.checkpoint_period,
            max_to_keep=mock_config.max_num_checkpoints_to_keep,
            enable_async_checkpointing=mock_config.async_checkpointing,
        ),
    )
    # AssertionError: CheckpointManager was called with extra/unexpected keyword argument 'item_handlers'

    💡 Root Cause Analysis & Context

    Confidence: high (Confirmed Cause)

    The failure was a codebase regression caused by PR #5208 ("yixuann-training-engine-support-ocdbt-zarr3").

    PR #5208 modified src/maxtext/training_engine/checkpointing.py to support OCDbt and Zarr3 formats by passing item_handlers to ocp.CheckpointManager. However, the unit test test_max_text_trainer_checkpoint_manager_init in tests/post_training/unit/maxtext_engine_test.py was not updated to expect this keyword argument, causing mock assertion failures in the CI run.

    🛠️ Recommended Fix

    Note: This has already been resolved on main via PR #5211 ("Fix CheckpointManager init unit test failure"), which updated the unit test to patch PyTreeCheckpointHandler, assert the presence of item_handlers in CheckpointManager's mock call, and verify it is instantiated with the expected OCDbt and Zarr3 storage configurations:

      @mock.patch("orbax.checkpoint.PyTreeCheckpointHandler")
      @mock.patch("orbax.checkpoint.CheckpointManager")
      def test_max_text_trainer_checkpoint_manager_init(self, mock_create_mgr, mock_handler):
        mock_config = self.setup_config(
            enable_checkpointing=True,
            checkpoint_storage_use_ocdbt=False,
            checkpoint_storage_use_zarr3=False,
        )
    
        _ = maxtext_engine.MaxTextTrainingEngine(mock_config)
        mock_create_mgr.assert_called_once_with(
            directory=mock_config.checkpoint_dir,
            options=ocp.CheckpointManagerOptions(
                save_interval_steps=mock_config.checkpoint_period,
                max_to_keep=mock_config.max_num_checkpoints_to_keep,
                enable_async_checkpointing=mock_config.async_checkpointing,
            ),
            item_handlers={
                "model_params": mock_handler.return_value,
                "optimizer_state": mock_handler.return_value,
                "accumulated_metrics": mock_handler.return_value,
                "accumulated_grads": mock_handler.return_value,
            },
        )
        self.assertEqual(mock_handler.call_count, 4)
        mock_handler.assert_has_calls(
            [mock.call(use_ocdbt=False, use_zarr3=False)] * 4,
        )
  28. github-actions commented on Sep 22, 2026

    @github-actions
    ContributorAuthor

    🤖 CI Failure Investigation Report

    I have analyzed the recent test failures in the CI pipeline and identified the following:

    🔍 What Failed

    • Job/Matrix: MaxText Package Tests
    • Failing Test: N/A (Environment/Setup Phase)
    • Error: The test logs in .gemini/failed_logs.txt are empty (0 bytes), indicating that the failure occurred during the pipeline setup, dependency installation, or environment provisioning phase before tests were collected or executed.

    🪵 Error Details & Stack Trace

    # No test traceback was recorded in .gemini/failed_logs.txt.
    # The scheduled build failed during the setup/provisioning phase of the "MaxText Package Tests" workflow.

    💡 Root Cause Analysis & Context

    Confidence: moderate

    1. Infrastructure/Environment Flake: Because the failed log file is entirely empty, the failure is highly likely related to transient infrastructure, environment configuration, or package installation issues (such as a timeout, network/GCS error, or container setup failure) rather than a codebase regression.
    2. Codebase Status: The most recent merge on main is PR Add basic RoPE implementation to m3 core #5325 ("Add basic RoPE implementation to m3 core"). I verified that the scaffolding tests for maxtext.m3 pass successfully under system Python 3.12. The mathematical and parity layers under rope_test.py are solid and correctly mirror the legacy implementation. No syntax, import, or logical regressions were introduced by this merge.

    🛠️ Recommended Fix

    As this is likely an infrastructure/environment flake, we recommend triggering a re-run of the failed workflow run (#13487) to verify if the issue resolves itself. If the failure persists, please inspect the workflow's setup steps (such as package installation via uv pip or container provisioning) for transient network or dependency resolution errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

build failedMaxText Package Tests Failed

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions