[workflow-analysis] Weekly Workflow Analysis: Active codex/copilot/claude CLI reliability incident (20% failure rate, driver-exit pattern)

[github-actions[bot]]

Overview

Analyzed 120 workflow runs across ~6.5 hours (2026-08-24 02:49–09:26 UTC) — the most recent, densely-sampled slice of the past week. This repo runs an extremely high volume of scheduled/triggered agentic workflows (292 workflow files configured, ~15–20 runs/hour observed), so a full 7-day exhaustive pull wasn't feasible in one pass; this sample is recency-weighted and surfaces an active reliability regression happening right now.

Overall: 24 failures / 120 runs (20% failure rate).

Key finding: an active, escalating "driver exit" incident (not an agent logic problem)

18 of 24 failures (75%) share the same signature: zero tokens, zero turns, zero errors logged — the engine CLI process crashed or exited before producing any model output. Only 2 failures were genuine agent-logic errors. This is an infrastructure/engine-CLI reliability issue, not a prompting or task-design problem.

It's also worsening within the observed window:

Time window (UTC)	codex runs	codex failures
03:26–05:49	9	0 (100% healthy)
06:14–07:57	6	5 (AI Moderator only)
08:55–08:59	3	3 (100% failing)

The AI Moderator workflow (codex engine) failed 5/5 times in-window (§32696507811, §32700858092, §32703386165, §32703902466, §32703917814) — all dying at the "Execute Codex CLI" step with 0 turns after making a handful of requests to api.openai.com/ab.chatgpt.com. Given other codex-driven workflows (Issue Monster, Sub-Issue Closer) succeeded in the same window, this doesn't look like a blanket outage — more like intermittent codex CLI crashes that are increasing in frequency toward the end of the sampled window.

The same driver-exit signature also hit copilot and claude, so it isn't codex-specific:

• Daily Go Test Parallelizer (copilot): failed 3/3 runs, every ~2h schedule tick (§32690847943, §32698289132, §32707485017) — dies at "Execute GitHub Copilot CLI" after only 2 requests to api.githubcopilot.com, 0 turns.
• Smoke Claude on Copilot (claude): failed 2/2 runs sampled.

Recommended action: file/escalate this as an active incident — check engine CLI versions, auth/token freshness, and any recent AWF runner or CLI-binary changes deployed around 2026-08-24 ~06:00 UTC, since that's when the failure rate visibly inflects.

Other notable items

• Cost outlier: "Terminal Stylist" (codex) burned 5.07M tokens (~$57 AIC) in a single run — the outlier is >2.5x the combined total of every other run in the sample. Worth checking whether this run got stuck in a retry/loop before its eventual failure.
• Cheap-model opportunity: the audit tool flagged AI Moderator's issue-response task as a candidate for a smaller model (e.g. gpt-4.1-mini, claude-haiku-4-5) instead of a frontier model — applies independent of the reliability fix.
• Slow runs (>20 min, no failure): Copilot Session Insights (45.5m), Safe Output Health Monitor (40.8m), Static Analysis Report (24.5m), Daily AgentRx Trace Optimizer (23.0m), Organization Health Report (22.2m) — worth a look if these are meant to run frequently on schedule, since long wall-clock time increases the chance of overlapping/queued runs.

View Details — engine & workflow breakdown

By engine (120 runs):

Engine	Runs	Failures	Failure rate
codex	21	9	43%
copilot	45	10	22%
claude	32	4	12%
pi	21	1	5%
crush	1	0	0%

Workflows with repeated failures in-window:

Workflow	Runs	Failures
AI Moderator	5	5
Daily Go Test Parallelizer	3	3
Smoke Claude on Copilot	2	2

Aggregate resource usage (120 runs, ~6.5h window):

• Total tokens: ~12.6M
• Total AIC (cost proxy): ~$6,065 — note the sample includes one 5M-token outlier run; excluding it, usage is far more typical
• Total Action minutes: 1,271
• Total GitHub API calls: 2,167

Next actions

1. Escalate the driver-exit pattern (0-token CLI crashes across codex/copilot/claude) as an active reliability incident — check auth/token expiry and any CLI/runner version changes deployed around 2026-08-24 06:00 UTC.
2. Investigate why AI Moderator and Daily Go Test Parallelizer are failing 100% of the time in this window specifically (vs. other workflows on the same engines succeeding).
3. Look into the "Terminal Stylist" 5M-token run for a possible retry loop or runaway cost.
4. Consider a cheaper model for AI Moderator's issue-response task once reliability is restored.

References:

• §32703917814 — AI Moderator failure (audited)
• §32707485017 — Daily Go Test Parallelizer failure (audited)
• §32708737765 — Terminal Stylist (codex, latest failure in the escalating window)

Generated by 🔍 Weekly Workflow Analysis · claude · agent · 159.3 AIC · ⌖ 6.33 AIC · ⊞ 5K · ◷

• expires on Aug 25, 2026, 1:48 AM UTC-08:00