Repository navigation
ci: schedule the nightly and RC pipelines for the E2E TPU quota window - #6030
Conversation
The MaxText E2E TPU tests are moving to the ubench-regression-tests queue on bodaborg-v6e-nap, which only has TPU quota from 20:00 to 24:00 Pacific time each day (b/568477549). With the current schedules, the nightly pipeline (00:00 UTC) submits its E2E TPU jobs at about 01:00-02:00 UTC and the RC pipeline (Mondays 06:00 UTC) at about 07:00-08:00 UTC, both outside the window. - Run the TPU Docker Images Pipeline daily, and the Release Candidate pipeline every Sunday, at 19:00 Pacific time. The first E2E TPU job is submitted about an hour after the scheduled time, when the window opens. The schedules use the America/Los_Angeles timezone, so they follow daylight saving time like the window does. - Bump actionlint to v1.7.12.25. v1.7.7 rejects the schedule timezone key, and v1.7.12 reports no findings on the existing workflows. - Update the TPU pipeline schedule in ci_docker_image_pipelines.md.
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request adjusts the scheduling of CI pipelines for TPU-based E2E tests to ensure they execute within the available quota window of the 'ubench-regression-tests' queue. By shifting these triggers to 19:00 Pacific time, the jobs will align with the daily quota availability, preventing submission failures. Additionally, the project's linting configuration was updated to accommodate the use of timezone-aware cron schedules. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Ignored Files
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
🤖 Hi @chiajunglien, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
There was a problem hiding this comment.
📋 Review Summary
Verdict: No significant issues found.
Inferred spec & validation
- Sources: PR description, code before/after.
- Problem: The daily TPU Docker images pipeline and weekly RC pipeline trigger E2E TPU jobs outside of the 20:00-24:00 PT quota window, resulting in jobs having zero quota.
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): Move the tpu_docker_images_pipeline schedule to daily at 19:00 PT. |
✅ | ❌ none |
R2 (stated): Move the rc_release schedule to Sunday at 19:00 PT. |
✅ | ❌ none |
R3 (stated): Use timezone: 'America/Los_Angeles' so the schedules follow daylight saving time exactly like the quota window. |
✅ | ❌ none |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Description
Move the nightly TPU Docker Images Pipeline and the weekly Release Candidate pipeline to 19:00 Pacific time, so their E2E TPU jobs start when the TPU quota window of the E2E test queue opens.
The MaxText E2E TPU tests are moving to the
ubench-regression-testsqueue onbodaborg-v6e-nap(see #5990). That queue only has TPU quota from 20:00 to 24:00 Pacific time each day, which is 03:00-07:00 UTC now and 04:00-08:00 UTC from November 1. Outside the window its quota is 0 (b/568477549). With the current schedules, both pipelines submit their E2E TPU jobs outside the window:Changes:
tpu_docker_images_pipeline.yml: run daily at 19:00 Pacific time. The first E2E TPU job is submitted about an hour after the scheduled time, when the window opens.rc_release.yml: run every Sunday at 19:00 Pacific time, which is Monday 02:00 or 03:00 UTC, a few hours earlier than now.timezone: 'America/Los_Angeles', so they follow daylight saving time like the window does..pre-commit-config.yaml: bump actionlint to v1.7.12.25. v1.7.7 rejects the scheduletimezonekey.docs/development/ci_docker_image_pipelines.md: update the TPU pipeline schedule.The GPU Docker Images Pipeline keeps its 00:00 UTC schedule.
Tests
pre-commit run --files <changed files>: codespell, mdformat, yamllint and actionlint v1.7.12.25 pass.element of "schedule" section must be mapping and must contain one key "cron"for each schedule entry with atimezone, and nothing else.UBENCH_WINDOW_START_HOUR=20,UBENCH_WINDOW_HOURS=4,SCHEDULE_TZ=PST8PDT). The submission times come from the job creation audit logs of the recent nightly and RC runs.Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.skip-ci-testsso only CPU/lint/docs checks run during review; a bot removes it after 2 approvals on a 1-commit PR to run TPU/GPU suites, or remove it yourself / ask a maintainer on fork PRs to run them sooner.)