Skip to content

fix(pool): grade a bare Fable 429 like any bare 429 instead of cooling every seat - #1135

Closed
panjinhe wants to merge 2 commits into
dofastted:mainfrom
panjinhe:pr/fable-bare-429
Closed

panjinhe wants to merge 2 commits into
dofastted:mainfrom
panjinhe:pr/fable-bare-429

Conversation

@panjinhe

Copy link
Copy Markdown
Contributor

Stacked on #1127, which adds gradeBare429. Only the last commit is new here: fix(pool): grade a bare Fable 429 like any bare 429….

Problem

Today a Fable 429 is always classified as fable_rate_limited, even when it has no 7d_oi window. That means a 60s Fable-family cooldown on the seat and failover to the next seat.

What we saw in production (1.4.112):

  • One session's /compact summary turn on claude-fable-5-2 got 429 {"type":"rate_limit_error","message":"Error"} with no ratelimit headers, on every seat it reached (5 seats).
  • Those same seats were serving that session's other Fable turns within seconds before and after.
  • Each client retry (about one per minute) cooled the Fable family on 3 more seats. The session's normal turns were then pushed onto other accounts, which caused all of the remaining session hops in the pool after sticky tightening (19 of 19).

Change

  • upstream-error-policy: only a 429 carrying the 7d_oi window is fable_rate_limited. A bare Fable 429 becomes fable_rate_limited_unknown, still scope model / fable family, with retrySameAccount: false.
  • failover-runner: fable_rate_limited_unknown goes through gradeBare429, the same as rate_limited_unknown:
    • the same request rejected on 2 accounts → rate_limited_request, stop, nothing cooled;
    • a single account only steps aside, and its Fable family cools once distinct sessions confirm.
    • Fable and account-wide bare-429 hits are counted under separate keys, so one never confirms the other.
  • schedule-state: the new reason is stateless, like fable_rate_limited.

A real Fable weekly limit that arrives as a bare 429 is still caught: a second session confirms it, and every 429 still asks the usage probe.

Tests

  • upstream-error-policy: bare Fable 429 → fable_rate_limited_unknown; the 7d_oi path is unchanged.
  • failover-runner:
    • bare Fable 429 on two accounts → stop, no cooldown;
    • second session → Fable cooldown;
    • Fable and account bare 429 do not cross-confirm.
  • failover-runner, upstream-error-policy, schedule-state, rate-limit-service: 155 pass.

… failed hops and classifier bodies in debug log

08:50 incident: one auto-mode classifier request got a bare 429 on 8 accounts in a row;
each 429 cooled its whole account for 60s and the pool went eligible=0.

- failover.max_accounts_per_request (0 = unlimited): hard stop after N accounts ran
- bare 429 on bare_429_request_scope_after (2) accounts of one request -> request-scope
  stop, nobody cooled
- a bare 429 cools the account only after bare_429_account_sessions (2) distinct
  sessions hit it within bare_429_window_ms; otherwise it only steps aside for the request
- failed hops (status, rate-limit headers, body, provider facts) go into the debug
  record as hop_errors
- classifier requests keep inbound/outbound bodies in debug mode
…g every seat

A Fable 429 without the 7d_oi window (body just "Error", no ratelimit headers)
was treated as a Fable window limit: 60s Fable cooldown on each seat and
failover across up to three seats. In production the provider answered such a
429 to one specific request (a /compact summary turn) on every seat, while the
same seats kept serving that session's other Fable turns. Each client retry
cooled the Fable family on more seats and pushed the session across accounts.

Now only a 429 carrying the 7d_oi window is fable_rate_limited. A bare one is
fable_rate_limited_unknown and goes through gradeBare429: the same request
rejected on two accounts stops as request scope and cools nobody; one account
cools its Fable family only once distinct sessions confirm it. Fable and
account-wide bare 429 hits are counted apart.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T14:51:54.850257Z dc99c38 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dc99c38cde

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +636 to +638
scope: 'request',
action: 'stop',
reason: 'rate_limited_request',

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Honor request-scoped 429s in final quota accounting

For native streaming errors with terminalState: 'incomplete', stopping after the second bare 429 now returns 429 with that account's attribution, rather than the previous mapped 502. The post-run path in handle-protocol.mjs:2176-2204 still calls isAccountQuotaExhausted(), which treats any non-pool 429 as account exhaustion. Consequently, AccountQuota.ingestHeaders() synthesizes a rejected 5h window and syncQuotaSchedule() restricts the second account across all models, despite the runner recording no cooldowns. Make final quota ingestion honor this request-scoped verdict so the new stop does not poison an otherwise healthy account.

Useful? React with 👍 / 👎.

Comment on lines +644 to +648
if (isFableModel(model)) {
return {
scope: 'model',
action: 'continue-and-cooldown',
reason: 'fable_rate_limited_unknown',

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Check rate-limit evidence before treating a Fable 429 as bare

If a Fable 429 includes anthropic-ratelimit-requests-remaining: 0 or explicit per-minute wording, this unconditional return prevents isShortWindowLimit() below from recognizing the evidence. The resulting fable_rate_limited_unknown enters gradeBare429(), which removes the first cooldown and stops after two accounts. With the first two accounts RPM-limited and a healthy third account, the parent reaches the third account and succeeds, whereas this commit returns 429 without trying it. Check the existing limit evidence before assigning the unknown classification.

Useful? React with 👍 / 👎.

Comment on lines +1263 to +1265
if (this.accountCapReached(ranAccounts)) {
releaseStickySlots()
return stopAtAccountCap(result, policy, ranAccounts.size, budget.hops, attribution())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Enforce the account cap before dispatching retry hops

With max_accounts_per_request: 1, an empty/incomplete first hop enters budget.allowSameUnit() and continues before this guard. That retry sets avoid to the current account, so the next selection explicitly prefers another account; if it succeeds, the request returns without ever checking the exceeded cap. This executes two distinct accounts despite a cap of one, and the exception retry branch has the same ordering. Enforce the limit before dispatching a newly selected account while still permitting retries on accounts already counted.

Useful? React with 👍 / 👎.

@dofastted

Copy link
Copy Markdown
Owner

感谢贡献。替代修复 #1165 已合入 main(b1842a588e)。模型名不证明 Fable 周窗耗尽。只有 7d_oi 等已知证据进入对应冷却,两个账号重复裸 429 停止换槽且不冷却。 此 PR 不再重复合并。

@dofastted dofastted closed this Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants