Repository navigation
Conversation
… failed hops and classifier bodies in debug log 08:50 incident: one auto-mode classifier request got a bare 429 on 8 accounts in a row; each 429 cooled its whole account for 60s and the pool went eligible=0. - failover.max_accounts_per_request (0 = unlimited): hard stop after N accounts ran - bare 429 on bare_429_request_scope_after (2) accounts of one request -> request-scope stop, nobody cooled - a bare 429 cools the account only after bare_429_account_sessions (2) distinct sessions hit it within bare_429_window_ms; otherwise it only steps aside for the request - failed hops (status, rate-limit headers, body, provider facts) go into the debug record as hop_errors - classifier requests keep inbound/outbound bodies in debug mode
…g every seat A Fable 429 without the 7d_oi window (body just "Error", no ratelimit headers) was treated as a Fable window limit: 60s Fable cooldown on each seat and failover across up to three seats. In production the provider answered such a 429 to one specific request (a /compact summary turn) on every seat, while the same seats kept serving that session's other Fable turns. Each client retry cooled the Fable family on more seats and pushed the session across accounts. Now only a 429 carrying the 7d_oi window is fable_rate_limited. A bare one is fable_rate_limited_unknown and goes through gradeBare429: the same request rejected on two accounts stops as request scope and cools nobody; one account cools its Fable family only once distinct sessions confirm it. Fable and account-wide bare 429 hits are counted apart.
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dc99c38cde
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| scope: 'request', | ||
| action: 'stop', | ||
| reason: 'rate_limited_request', |
There was a problem hiding this comment.
Honor request-scoped 429s in final quota accounting
For native streaming errors with terminalState: 'incomplete', stopping after the second bare 429 now returns 429 with that account's attribution, rather than the previous mapped 502. The post-run path in handle-protocol.mjs:2176-2204 still calls isAccountQuotaExhausted(), which treats any non-pool 429 as account exhaustion. Consequently, AccountQuota.ingestHeaders() synthesizes a rejected 5h window and syncQuotaSchedule() restricts the second account across all models, despite the runner recording no cooldowns. Make final quota ingestion honor this request-scoped verdict so the new stop does not poison an otherwise healthy account.
Useful? React with 👍 / 👎.
| if (isFableModel(model)) { | ||
| return { | ||
| scope: 'model', | ||
| action: 'continue-and-cooldown', | ||
| reason: 'fable_rate_limited_unknown', |
There was a problem hiding this comment.
Check rate-limit evidence before treating a Fable 429 as bare
If a Fable 429 includes anthropic-ratelimit-requests-remaining: 0 or explicit per-minute wording, this unconditional return prevents isShortWindowLimit() below from recognizing the evidence. The resulting fable_rate_limited_unknown enters gradeBare429(), which removes the first cooldown and stops after two accounts. With the first two accounts RPM-limited and a healthy third account, the parent reaches the third account and succeeds, whereas this commit returns 429 without trying it. Check the existing limit evidence before assigning the unknown classification.
Useful? React with 👍 / 👎.
| if (this.accountCapReached(ranAccounts)) { | ||
| releaseStickySlots() | ||
| return stopAtAccountCap(result, policy, ranAccounts.size, budget.hops, attribution()) |
There was a problem hiding this comment.
Enforce the account cap before dispatching retry hops
With max_accounts_per_request: 1, an empty/incomplete first hop enters budget.allowSameUnit() and continues before this guard. That retry sets avoid to the current account, so the next selection explicitly prefers another account; if it succeeds, the request returns without ever checking the exceeded cap. This executes two distinct accounts despite a cap of one, and the exception retry branch has the same ordering. Enforce the limit before dispatching a newly selected account while still permitting retries on accounts already counted.
Useful? React with 👍 / 👎.
|
感谢贡献。替代修复 #1165 已合入 main(b1842a588e)。模型名不证明 Fable 周窗耗尽。只有 7d_oi 等已知证据进入对应冷却,两个账号重复裸 429 停止换槽且不冷却。 此 PR 不再重复合并。 |
Stacked on #1127, which adds
gradeBare429. Only the last commit is new here:fix(pool): grade a bare Fable 429 like any bare 429….Problem
Today a Fable 429 is always classified as
fable_rate_limited, even when it has no7d_oiwindow. That means a 60s Fable-family cooldown on the seat and failover to the next seat.What we saw in production (1.4.112):
/compactsummary turn onclaude-fable-5-2got429 {"type":"rate_limit_error","message":"Error"}with no ratelimit headers, on every seat it reached (5 seats).Change
upstream-error-policy: only a 429 carrying the7d_oiwindow isfable_rate_limited. A bare Fable 429 becomesfable_rate_limited_unknown, still scopemodel/ fable family, withretrySameAccount: false.failover-runner:fable_rate_limited_unknowngoes throughgradeBare429, the same asrate_limited_unknown:rate_limited_request, stop, nothing cooled;schedule-state: the new reason is stateless, likefable_rate_limited.A real Fable weekly limit that arrives as a bare 429 is still caught: a second session confirms it, and every 429 still asks the usage probe.
Tests
upstream-error-policy: bare Fable 429 →fable_rate_limited_unknown; the7d_oipath is unchanged.failover-runner:failover-runner,upstream-error-policy,schedule-state,rate-limit-service: 155 pass.