Skip to content

ci: schedule the nightly and RC pipelines for the E2E TPU quota window - #6030

Merged
copybara-service[bot] merged 1 commit into
mainfrom
emma/e2e-schedule-quota-window
Nov 2, 2026
Merged

copybara-service[bot] merged 1 commit into
mainfrom
emma/e2e-schedule-quota-window

Conversation

@chiajunglien

Copy link
Copy Markdown
Collaborator

Description

Move the nightly TPU Docker Images Pipeline and the weekly Release Candidate pipeline to 19:00 Pacific time, so their E2E TPU jobs start when the TPU quota window of the E2E test queue opens.

The MaxText E2E TPU tests are moving to the ubench-regression-tests queue on bodaborg-v6e-nap (see #5990). That queue only has TPU quota from 20:00 to 24:00 Pacific time each day, which is 03:00-07:00 UTC now and 04:00-08:00 UTC from November 1. Outside the window its quota is 0 (b/568477549). With the current schedules, both pipelines submit their E2E TPU jobs outside the window:

  • the nightly pipeline (00:00 UTC) at about 01:00-02:00 UTC,
  • the RC pipeline (Mondays 06:00 UTC) at about 07:00-08:00 UTC.

Changes:

  • tpu_docker_images_pipeline.yml: run daily at 19:00 Pacific time. The first E2E TPU job is submitted about an hour after the scheduled time, when the window opens.
  • rc_release.yml: run every Sunday at 19:00 Pacific time, which is Monday 02:00 or 03:00 UTC, a few hours earlier than now.
  • Both schedules use timezone: 'America/Los_Angeles', so they follow daylight saving time like the window does.
  • On Sundays the nightly and the RC E2E tests both run in that window.
  • .pre-commit-config.yaml: bump actionlint to v1.7.12.25. v1.7.7 rejects the schedule timezone key.
  • docs/development/ci_docker_image_pipelines.md: update the TPU pipeline schedule.

The GPU Docker Images Pipeline keeps its 00:00 UTC schedule.

Tests

  • pre-commit run --files <changed files>: codespell, mdformat, yamllint and actionlint v1.7.12.25 pass.
  • actionlint v1.7.12 reports no findings on all workflows. v1.7.7 reports element of "schedule" section must be mapping and must contain one key "cron" for each schedule entry with a timezone, and nothing else.
  • actionlint v1.7.12 rejects an invalid timezone name, so it checks the key rather than ignoring it.
  • The window comes from the queue's quota schedule daemon (UBENCH_WINDOW_START_HOUR=20, UBENCH_WINDOW_HOURS=4, SCHEDULE_TZ=PST8PDT). The submission times come from the job creation audit logs of the recent nightly and RC runs.
  • Scheduled triggers cannot run on a PR. The first runs after merge should start at about 19:00 Pacific time.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable. (Note: New PRs get skip-ci-tests so only CPU/lint/docs checks run during review; a bot removes it after 2 approvals on a 1-commit PR to run TPU/GPU suites, or remove it yourself / ask a maintainer on fork PRs to run them sooner.)
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

The MaxText E2E TPU tests are moving to the ubench-regression-tests
queue on bodaborg-v6e-nap, which only has TPU quota from 20:00 to 24:00
Pacific time each day (b/568477549). With the current schedules, the
nightly pipeline (00:00 UTC) submits its E2E TPU jobs at about
01:00-02:00 UTC and the RC pipeline (Mondays 06:00 UTC) at about
07:00-08:00 UTC, both outside the window.

- Run the TPU Docker Images Pipeline daily, and the Release Candidate
  pipeline every Sunday, at 19:00 Pacific time. The first E2E TPU job is
  submitted about an hour after the scheduled time, when the window
  opens. The schedules use the America/Los_Angeles timezone, so they
  follow daylight saving time like the window does.
- Bump actionlint to v1.7.12.25. v1.7.7 rejects the schedule timezone
  key, and v1.7.12 reports no findings on the existing workflows.
- Update the TPU pipeline schedule in ci_docker_image_pipelines.md.
@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request adjusts the scheduling of CI pipelines for TPU-based E2E tests to ensure they execute within the available quota window of the 'ubench-regression-tests' queue. By shifting these triggers to 19:00 Pacific time, the jobs will align with the daily quota availability, preventing submission failures. Additionally, the project's linting configuration was updated to accommodate the use of timezone-aware cron schedules.

Highlights

  • Pipeline Schedule Adjustment: Updated the nightly TPU Docker Images and weekly Release Candidate pipelines to trigger at 19:00 Pacific time to align with the E2E TPU quota window.
  • Tooling Update: Bumped actionlint to v1.7.12.25 in .pre-commit-config.yaml to support the timezone key in GitHub Actions schedules.
  • Documentation Update: Updated the CI documentation to reflect the new scheduling for the TPU Docker image pipelines.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Ignored Files
  • Ignored by pattern: .github/workflows/** (2)
    • .github/workflows/rc_release.yml
    • .github/workflows/tpu_docker_images_pipeline.yml
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions github-actions Bot added the skip-ci-tests Add this label to the PR to skip running CI tests label Oct 29, 2026
@chiajunglien
chiajunglien requested a review from shralex as a code owner October 29, 2026 06:58
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @chiajunglien, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: No significant issues found.

Inferred spec & validation
  • Sources: PR description, code before/after.
  • Problem: The daily TPU Docker images pipeline and weekly RC pipeline trigger E2E TPU jobs outside of the 20:00-24:00 PT quota window, resulting in jobs having zero quota.
Spec item Implementation Validating test
R1 (stated): Move the tpu_docker_images_pipeline schedule to daily at 19:00 PT. ✅ ❌ none
R2 (stated): Move the rc_release schedule to Sunday at 19:00 PT. ✅ ❌ none
R3 (stated): Use timezone: 'America/Los_Angeles' so the schedules follow daylight saving time exactly like the quota window. ✅ ❌ none

@codecov

codecov Bot commented Oct 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@SurbhiJainUSC SurbhiJainUSC added pull ready and removed skip-ci-tests Add this label to the PR to skip running CI tests labels Nov 2, 2026
@copybara-service
copybara-service Bot merged commit 47430c6 into main Nov 2, 2026
156 of 159 checks passed
@copybara-service
copybara-service Bot deleted the emma/e2e-schedule-quota-window branch November 2, 2026 21:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants