early-stop: unweighted stop decisions, undocumented live_verdict contract, no step budget

[akshaylive]

Summary

Three follow-ups to the stop_early / EarlyStopWatcher design (src/coder_eval/orchestration/early_stop.py), captured during a design review, plus one new idea worth designing properly: a weighted-score-based break condition.

1. Stop decisions and the pass/fail gate ignore weight

EarlyStopWatcher's stop rule (_evaluate/_fire) and EvaluationResult.armed_criteria_passed operate purely on ternary LiveVerdict (pass/fail/undecided) and each criterion's own pass_threshold. Neither the live stop decision nor the early-stopped gate consults criterion.weight at all:

• A single fail-armed criterion — regardless of how trivial its weight is relative to the rest of the suite — can end the entire run for every other criterion the instant the defer-gate opens.
• armed_criteria_passed is an unweighted all(...) over the armed subset; one low-weight criterion failing its own pass_threshold fails the whole gate.
• Unarmed (possibly higher-weight) criteria still get scored against the truncated trajectory post-hoc via weighted_score, but that score plays no role in the stop decision or the gate — it's purely advisory and easy to miss.

Ask: decide whether/how severity (e.g. weight) should factor into the stop decision, so a trivial armed criterion can't unilaterally truncate a run that a more important unarmed criterion would otherwise have passed given more turns.

2. live_verdict monotonicity + determinism is an undocumented, unenforced contract

The correctness of the deferred fail-stop and _prev_verdicts flip-attribution silently depends on every armed live_verdict being:

• a pure/deterministic function of turn_records, and
• monotonic — once it returns "pass"/"fail" for some trajectory prefix, it must return the same verdict for any longer trajectory.

This currently holds only by construction/convention in the two existing checkers (skill_triggered.py, command_executed.py — audited, both verified sound). Nothing on BaseCriterion.live_verdict/LiveVerdict states this as a requirement, and CE025 only checks live_stop_polarities/live_verdict override pairing, not monotonicity. A future third criterion (in-tree or third-party plugin) implementing live_verdict non-monotonically would compile, pass lint, and silently corrupt the stop logic.

Ask: document the contract explicitly on the abstract method/protocol (docstring at minimum; consider a property-style test or lint rule if feasible).

3. No step budget on "still undecided"

There's currently no cap on how long the watcher will keep evaluating a run that never decides — an armed-but-undecided suite just rides passively along until the run ends or hits max_turns normally, with EarlyStopWatcher doing nothing useful the whole time.

Ask: add an N-step (tool-call) budget after which, if all armed criteria are still "undecided", the run breaks rather than continuing silently. Needs a decision on what "breaks" means here — does it fail the run, fall back to a normal full run to max_turns, or something else?

4. New idea to design: run_limits config for a weighted-score-triggered break

Distinct from the above — should there be a config that breaks the run early based on the partial weighted score itself, not just individual criterion verdicts?

Rough direction (needs real design, not committing to this yet):

• The current design's safety property is that a stop can never be wrong — live_verdict only fires when the outcome is monotonically locked in.
• A naive "stop once weighted score crosses threshold X" breaks that property, since scores from continuous/fractional criteria (e.g. llm_judge, reference_comparison) aren't necessarily monotonic over a partial trajectory.
• A safer version: compute a bound, not the live score itself — treat every criterion still undecided/unscored as contributing its worst case (0) for a floor bound, or its best case (1.0 × weight) for a ceiling bound. If the floor already meets/exceeds the pass threshold → guaranteed pass, safe to stop early. If the ceiling can't reach the threshold even in the best case → guaranteed fail, safe to stop early. This preserves the "never a false stop" invariant the rest of the design relies on.
• Would need a way to say "not yet scoreable, but if it were, worst/best case is X" per criterion — more invasive than the current ternary live_verdict protocol.

Ask: figure out whether the bound-based approach above is the right shape, or something simpler/different, before implementing anything.