fix: bound the optimizer's tool calls from the harness, not the prompt

The two commits under this PR diagnosed the failure correctly and then fixed it
in the one place that cannot enforce anything. The optimizer's stdout reaches
harbor as one long-lived stream, a harness flushes a command's output only when
that command returns, and an idle stream gets reaped while the machine, the
connection and the sandbox stay healthy. So "how long may one tool call run" and
"how long may the stream go silent" are the same question, and the outer trial
is not retried when the answer is too long.

Asking the optimizer to wait in bounded steps is the right shape at the wrong
layer. The evidence is in this branch's own history: the recipe shipped in the
first commit spun forever on both arms and had to be replaced in the second. A
prompt is advisory, a model reconstructing the loop from memory reintroduces the
failure, and nothing in the system notices when it does.

Both harnesses already expose the bound as a setting, so set it.
HARNESS_TOOL_TIMEOUT_SECONDS = 300 goes out per harness through harbor's --ae
seam: opencode reads OPENCODE_EXPERIMENTAL_BASH_DEFAULT_TIMEOUT_MS, claude-code
reads BASH_DEFAULT_TIMEOUT_MS and BASH_MAX_TIMEOUT_MS. Harnesses with no
verified knob (codex, mini-swe-agent) are sent nothing rather than a variable
they ignore. It is placed ahead of the build's own agent_env so a build can
raise or drop the cap by naming the same variable, since harbor's parse_env_vars
keeps the last value for a key.

opencode's is a default only: packages/opencode/src/tool/shell.ts resolves
`flags.bashDefaultTimeoutMs ?? 2 * 60 * 1000` and then `params.timeout ??
defaultTimeoutMs` with no clamp, so a model naming its own timeout still escapes
it. It covers the case that actually killed the run, where the instruction said
to let the call block and the model therefore never named one, and it lowers the
number quoted in the tool description the model reads. claude-code's MAX is a
true ceiling.

The cap alone would be worse than nothing: a five-minute kill on a thirty-minute
`evals run` hands the model opencode's own "retry with a larger timeout"
message, which restores the silence and burns an evaluation. So `evals run` now
always starts a job and polls it internally on a 240s bound, 255s worst case,
returning the job record with its job_id when the bound expires. `evals wait`
resumes, and the evaluation keeps running in the sidecar throughout. Same
evaluation as before: the sidecar drives POST /eval and POST /eval/jobs through
the same _execute_tracked_job, same budget, same SidecarEvaluationResult. The
detached path records failures on the job record instead of raising, so
_await_evaluation_job reads the reason back off it and raises a ClickException,
preserving today's non-zero exit.

The instruction template loses the recipe, the contract essay and the incident
narrative that the two earlier commits added, about 45 lines, and states the
behavior in four. The number it quotes is read from WAIT_TIMEOUT_SECONDS through
the template context rather than restated, so prompt and code cannot drift.
SKILL.md and docs/guide.md follow.

test_harbor_cli_builds_canonical_selection pinned the old contract that a plain
run posts to /eval; it now pins /eval/jobs. That was the only existing test the
change broke.

Test plan: 490 passed, 15 skipped (485 before, five new tests: the per-harness
bound including the HARNESS_TOOL_TIMEOUT_SECONDS > WAIT_TIMEOUT_SECONDS
ordering, a build overriding the cap through agent_env, and the three
_await_evaluation_job paths, bound expiry, completion and failure). Credentials
must be exported or five unrelated tests fail on compiler.py credential
validation.

Not verified end to end. The failure is probabilistic, the same stream in the
dead run survived a 24.5-minute gap and then died after ten, so a live A/B would
need many full optimization runs to say anything. What is verified is that the
variables reach the harness and that the CLI returns inside the cap.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: shehabyasser-scale

vero/docs/guide.md

@@ -96,11 +96,15 @@ build.yaml --output task` compiles without running, for inspection.
 > Harbor constructs the agent; a raw `harbor run` would let the agent adapter
 > read the upstream key from its own host process first.
 
-Inside the container the agent evaluates candidates with `evals run
---detach`, then `evals status JOB` / `evals result JOB` / `evals status` (via `VERO_EVAL_URL`).
-Detached evaluations are **durable jobs** — the candidate version is captured
-before the command returns, so ending the agent process can't lose or race a
-running measurement.
+Inside the container the agent evaluates candidates with `evals run` (via
+`VERO_EVAL_URL`), which waits for the result and, if the evaluation outlives
+that wait's bound, hands back a `job_id` for `evals wait JOB`; `--detach`
+returns the id immediately instead, for evaluating concurrently. Either way the
+evaluation is a **durable job**: the candidate version is captured before the
+command returns, so ending the agent process can't lose or race a running
+measurement. The bound matters because vero also caps how long one of the
+optimizer's tool calls may run (`HARNESS_TOOL_TIMEOUT_SECONDS`), so an `evals`
+call has to return on its own terms rather than be killed mid-evaluation.
 
 ### How the boundaries hold
 

vero/src/vero/evals_cli.py

@@ -26,6 +26,25 @@
 CONTEXT_DIRECTORY = ".evals"
 _CELL_WIDTH = 48
 
+# How long a blocking `evals run` / `evals wait` waits before returning what it
+# has so far.
+#
+# Not ergonomics: the optimizer's whole process is read through one long-lived
+# stdout stream, and an agent harness only flushes a command's output when that
+# command *returns*. A call that sits silent long enough for the network path to
+# reap that stream kills the trial, and the outer trial is not retried. Observed
+# 2026-07-31: a cell died at 71 minutes, 9m57s into a single silent wait,
+# discarding a candidate that had already scored 0.1224 on 49 validation cases.
+# Returning on a bound is what keeps the stream alive, and each return costs one
+# line of output.
+#
+# Deliberately below HARNESS_TOOL_TIMEOUT_SECONDS (`vero/harbor/cli.py`), the cap
+# vero sets on the optimizer's harness: these commands must return on their own
+# terms, handing back a job id that can be waited on again, rather than be killed
+# by the harness and hand the model a "retry with a larger timeout" message.
+WAIT_TIMEOUT_SECONDS = 240.0
+WAIT_POLL_INTERVAL_SECONDS = 15.0
+
 
 # --------------------------------------------------------------------------
 # Context discovery and shared helpers
@@ -301,10 +320,11 @@ def evals() -> None:
     resources), `candidates/` (prior program versions), and `plan.json`
     (what you may evaluate, and remaining budget).
 
-    Typical loop: `evals plan` -> edit + commit -> `evals run` (blocks and
-    returns the result) -> `evals diff BASELINE CANDIDATE` ->
-    `evals cases ID --sort score` -> `evals trace ID CASE`. Add `--detach` only
-    to run several evaluations at once, then poll `evals status JOB`.
+    Typical loop: `evals plan` -> edit + commit -> `evals run` (waits for the
+    result, or returns a `job_id` to `evals wait` on) -> `evals diff BASELINE
+    CANDIDATE` -> `evals cases ID --sort score` -> `evals trace ID CASE`. Add
+    `--detach` only to run several evaluations at once, then poll
+    `evals status JOB`.
     """
 
 
@@ -384,34 +404,41 @@ def status_command(job_id):
 @click.argument("job_id")
 @click.option(
     "--poll-interval",
-    default=15.0,
+    default=WAIT_POLL_INTERVAL_SECONDS,
     show_default=True,
     type=click.FloatRange(min=1),
     help="Seconds between status polls.",
 )
 @click.option(
     "--timeout",
+    default=WAIT_TIMEOUT_SECONDS,
+    show_default=True,
     type=click.FloatRange(min=0),
-    help="Optional max seconds to wait. On expiry, print the current "
-    "(still-running) status and exit 0 so you can simply call `evals wait` "
-    "again. Default: wait until the job finishes.",
+    help="Max seconds to wait. On expiry, print the current (still-running) "
+    "status and exit 0, so calling `evals wait` again resumes the wait. The "
+    "default is bounded on purpose; raising it risks the run (see the docstring).",
 )
 def wait_command(job_id, poll_interval, timeout):
-    """Block until a detached job finishes, then print its result.
+    """Wait for a detached job, then print its result.
 
     The blocking companion to `evals run --detach`: one call that waits, so you
-    never hand-roll a poll loop. Idempotent — safe to call again if it returns
-    while the job is still running (only happens when --timeout is set).
+    never hand-roll a poll loop. Idempotent, so it is always safe to call again
+    when it returns while the job is still running.
+
+    The wait is *bounded* by default rather than open-ended. A call that returns
+    nothing for tens of minutes can get the whole run killed (see
+    WAIT_TIMEOUT_SECONDS), and re-entering costs one line of output, so the
+    bound is the default rather than something to opt into.
     """
     request = _sidecar_request()
     terminal = {"complete", "failed", "cancelled"}
-    deadline = None if timeout is None else time.monotonic() + timeout
+    deadline = time.monotonic() + timeout
     while True:
         job = request("GET", f"/eval/jobs/{job_id}")
         status = job.get("status") if isinstance(job, dict) else None
         if status in terminal:
             break
-        if deadline is not None and time.monotonic() >= deadline:
+        if time.monotonic() >= deadline:
             click.echo(json.dumps(_enrich_job(job), indent=2))
             return
         time.sleep(poll_interval)

vero/src/vero/harbor/build/compiler.py

@@ -15,6 +15,7 @@
 from importlib.metadata import version as distribution_version
 from pathlib import Path, PurePosixPath
 
+from vero.evals_cli import WAIT_TIMEOUT_SECONDS
 from vero.evaluation import (
     EvaluationBudget,
     EvaluationLimits,
@@ -787,6 +788,9 @@ def compile_harbor_task(
         ],
         "exhaust_budget": config.instruct_exhaust_budget,
         "disclose_budget": config.disclose_budget,
+        # Read from the CLI constant rather than restated here, so the number the
+        # instruction quotes cannot drift from the one `evals run` enforces.
+        "wait_timeout_seconds": WAIT_TIMEOUT_SECONDS,
         "build_timeout": config.build_timeout_seconds,
         "verifier_timeout": (
             config.verifier_timeout_seconds or max(1, int(config.timeout_seconds))

vero/src/vero/harbor/build/templates/instruction.md.j2

@@ -25,11 +25,11 @@ hidden final evaluation. The trusted evaluation sidecar owns the cases, scoring,
      --partition {{ selection_partition }}
    ```
 
-   An evaluation can take many minutes. For a **short** one, let the call block
-   and read the result it returns. For anything longer than a few minutes, use
-   the bounded wait loop below instead: a single call that blocks in silence for
-   half an hour can get your run killed. Iterate cheaply on a subset first
-   (`--start 0 --stop N`, or repeated `--case-id ID`).
+   An evaluation can take many minutes. `evals run` waits for it and prints the
+   result. If it is still running after {{ wait_timeout_seconds | int }}s the
+   command returns the job record instead, carrying a `job_id`; the evaluation
+   keeps running and `evals wait JOB_ID` resumes the wait. Iterate cheaply on a
+   subset first (`--start 0 --stop N`, or repeated `--case-id ID`).
    {% if seed_supported %}
    Pass `--seed N` to reproduce a noisy comparison exactly.
    {% else %}
@@ -47,50 +47,19 @@ hidden final evaluation. The trusted evaluation sidecar owns the cases, scoring,
    and `evals diff OLD NEW`. Do not re-run an evaluation to see a number you
    truncated — look it up.
 
-   Add `--detach` to run several evaluations at once, and to keep any long one
-   from going silent: it returns a `job_id` immediately instead of blocking.
-   Then wait for it in **bounded** steps rather than one open-ended call:
-
-   ```bash
-   evals run --detach --backend ... --evaluation-set ... --partition ...
-   # -> prints a job_id
-   evals wait JOB_ID --timeout 300
-   # -> returns within 5 minutes whether or not the job is done; call it again
-   #    while it is still running
-   ```
-
-   `evals wait --timeout N` returns when the job reaches a terminal state OR
-   after N seconds, whichever comes first, and exits 0 either way, so calling it
-   again is always safe. A job is terminal when `evals status JOB_ID` reports
-   `complete`, `failed` or `cancelled`. **Check the status; do not loop on the
-   text of the wait output.** On success `evals wait` prints the evaluation
-   *result*, not the job record, so a loop that waits for the word "complete" in
-   that output never terminates, and on a `failed` or `cancelled` job every
-   further wait returns instantly, turning the same loop into a busy spin that
-   burns the rest of your run. Plain `evals wait JOB_ID` with no `--timeout`
-   blocks until the job finishes, which is fine for a short evaluation and is
-   the thing to avoid for a long one.
-
-   **Why bounded, and not one long block.** Your process is read through a
-   single long-lived stdout stream, and a harness only flushes a command's
-   output when that command *returns*. A wait that blocks for half an hour
-   therefore emits nothing for half an hour, and a stream idle that long can be
-   torn down by the network path even though the machine, the connection and the
-   sandbox are all healthy. When that happens the run dies with
-   `StreamTerminatedError` / "Connection lost", the outer trial is not retried,
-   and the whole optimization is lost. Observed 2026-07-31: a cell
-   died at 71 minutes, 9m57s into a silent wait, having already earned a 0.1224
-   validation score. A `--timeout 300` loop caps that silence at five minutes
-   and costs nothing, because each return is just one more line of output.
+   Add `--detach` **only** to run several evaluations at once: it returns a
+   `job_id` immediately without waiting. `evals wait JOB_ID` then waits for one
+   on the same bound, and `evals status JOB_ID` reports its state and elapsed
+   time without waiting at all. A job is finished when its status is `complete`,
+   `failed` or `cancelled`; any other status means it is still running, so wait
+   on it again.
 
    **Run every `evals` call in the foreground.** You are a single-shot headless
    run: nothing exists to deliver a notification or wake you later. If you put a
    long call in a background task, schedule a wake-up, or say you will "report
    back when it finishes", the run simply ends there and whatever you have not
-   submitted is lost. The bounded wait loop above is still foreground: it blocks
-   the whole time, it just returns and re-enters every few minutes instead of
-   sitting silent, so it satisfies this rule rather than bending it. A call that
-   blocks for a few minutes is working correctly — let it block. To wait on two
+   submitted is lost. A call that waits for minutes and hands back a
+   still-running job is working correctly: wait on it again. To wait on two
    jobs, wait out the first, then the second. Do not claim an improvement before
    its comparison baseline has been scored on the same cases.
 

vero/src/vero/harbor/cli.py

@@ -11,12 +11,18 @@
 import subprocess
 import sys
 import tempfile
+import time
 import urllib.error
 import urllib.request
 from pathlib import Path
 
 import click
 
+from vero.evals_cli import (
+    WAIT_POLL_INTERVAL_SECONDS,
+    WAIT_TIMEOUT_SECONDS,
+    _enrich_job,
+)
 from vero.evaluation import (
     CaseIds,
     CaseRange,
@@ -231,6 +237,42 @@ def _load_env_file(path: Path) -> dict[str, str]:
 # the gateway token cap are the intended limits.
 OPENCODE_STEP_LIMIT = 1000
 
+# The ceiling vero puts on a single optimizer tool call, in seconds.
+#
+# The optimizer's stdout reaches harbor as one long-lived stream, and an agent
+# harness only flushes output when a tool call *returns*. "How long may one tool
+# call run" is therefore the same question as "how long may that stream go
+# silent", and an idle stream gets reaped by the network path while the machine,
+# the connection and the sandbox all stay healthy. The outer trial is not
+# retried, so the whole optimization goes with it: on 2026-07-31 a cell died at
+# 71 minutes, 9m57s into one silent call, discarding a candidate that had
+# already scored 0.1224 on 49 validation cases.
+#
+# Configured on the harness rather than requested in the instruction, because
+# the instruction is advisory and this is not. Telling the optimizer to wait in
+# bounded steps leaves it free to ignore the advice, and free to reconstruct the
+# loop wrongly. Sits above WAIT_TIMEOUT_SECONDS so the evals CLI always returns
+# first on its own terms; the cap is the backstop for everything else the
+# optimizer runs.
+HARNESS_TOOL_TIMEOUT_SECONDS = 300
+
+# How each harness spells "bound one tool call". Only knobs verified in the
+# harness's own source or docs are listed: a harness missing here keeps its own
+# default rather than being sent a variable it silently ignores.
+_TOOL_TIMEOUT_ENVIRONMENT: dict[str, tuple[str, ...]] = {
+    # opencode reads a *default* only. `packages/opencode/src/tool/shell.ts`
+    # resolves `flags.bashDefaultTimeoutMs ?? 2 * 60 * 1000` and then
+    # `params.timeout ?? defaultTimeoutMs` with no clamp, so a model that names
+    # its own timeout still escapes the bound. It covers the case that actually
+    # killed the run -- the instruction said to let the call block, so the model
+    # never named one -- and it lowers the default quoted in the tool
+    # description the model reads.
+    "opencode": ("OPENCODE_EXPERIMENTAL_BASH_DEFAULT_TIMEOUT_MS",),
+    # claude-code takes both, and its MAX is a true ceiling: a per-call timeout
+    # from inside the conversation cannot raise it.
+    "claude-code": ("BASH_DEFAULT_TIMEOUT_MS", "BASH_MAX_TIMEOUT_MS"),
+}
+
 # Harnesses that drive the model through litellm rather than a provider SDK.
 # litellm reads the base URL as <PROVIDER>_API_BASE; the SDKs read
 # <PROVIDER>_BASE_URL. vero sets the SDK names, so a litellm-based harness sees no
@@ -375,6 +417,28 @@ def _opencode_gateway_args(agent: str, model: str | None, task: Path) -> list[st
     return ["--ak", f"opencode_config={json.dumps(payload, separators=(',', ':'))}"]
 
 
+def _agent_tool_timeout_args(agent: str) -> list[str]:
+    """Bound one optimizer tool call, so no single call can idle the stream.
+
+    The mechanism the previous fix reached for was the instruction: it told the
+    optimizer to wait in bounded steps instead of one open-ended block. That is
+    the right shape and the wrong layer. A prompt cannot enforce anything, the
+    recipe it shipped had to be corrected twice in review, and a model that
+    reconstructs the loop from memory reintroduces the failure. The harnesses
+    already expose the bound as a setting; set it.
+
+    Returns `--ae NAME=VALUE` pairs, which harbor merges into the scoped exec env
+    wrapping the agent's run phase. Values are milliseconds, the unit every
+    harness here uses.
+    """
+
+    milliseconds = int(HARNESS_TOOL_TIMEOUT_SECONDS * 1000)
+    arguments: list[str] = []
+    for name in _TOOL_TIMEOUT_ENVIRONMENT.get(agent, ()):
+        arguments.extend(["--ae", f"{name}={milliseconds}"])
+    return arguments
+
+
 def _outer_app_name_args(
     environment: str, config_name: str, extra: tuple[str, ...]
 ) -> list[str]:
@@ -806,6 +870,10 @@ def run_command(config_path, agent, model, environment, params, env_file, extra)
         ]
         if model is not None:
             command.extend(["-m", model])
+        # Ahead of the build's own agent env, so a build can raise or drop the
+        # cap by naming the same variable: harbor keeps the last value for a key,
+        # and a vero default must not silently outrank an explicit choice.
+        command.extend(_agent_tool_timeout_args(agent))
         # Forward the build's declared agent env to the optimizer agent's shell.
         # Harbor's `--ae KEY=VALUE` populates the agent's extra_env, which harbor
         # injects into the agent's setup/install exec (scoped_exec_env). Sorted
@@ -887,6 +955,43 @@ def inference_gateway_command(config_path, host, port):
     serve_inference_gateway(config_path=config_path, host=host, port=port)
 
 
+def _await_evaluation_job(job: dict, timeout: float = WAIT_TIMEOUT_SECONDS) -> dict:
+    """Wait out a started evaluation job, bounded, and return what to print.
+
+    A blocking `POST /eval` is one HTTP call that can take half an hour and
+    prints nothing until it returns, which is precisely the silence that killed
+    an optimization on 2026-07-31. The sidecar drives both entry points through
+    the same tracked job (`Sidecar._execute_tracked_job`), so starting a job and
+    polling it is the same evaluation, the same budget and the same
+    `SidecarEvaluationResult` -- only interruptible.
+
+    Returns the evaluation result once the job completes, or the job record when
+    the bound expires first, in which case the caller re-enters with
+    `evals wait JOB_ID` and the evaluation keeps running in the sidecar
+    regardless. A failed or cancelled job raises, so a bounded run still exits
+    non-zero carrying the sidecar's own reason, exactly as the blocking call did.
+    """
+
+    job_id = job.get("job_id") if isinstance(job, dict) else None
+    if not job_id:
+        return job
+    terminal = {"complete", "failed", "cancelled"}
+    deadline = time.monotonic() + timeout
+    while True:
+        status = job.get("status")
+        if status in terminal:
+            break
+        if time.monotonic() >= deadline:
+            return _enrich_job(job)
+        time.sleep(WAIT_POLL_INTERVAL_SECONDS)
+        job = _request("GET", f"/eval/jobs/{job_id}")
+    if status == "complete":
+        return _request("GET", f"/eval/jobs/{job_id}/result")
+    raise click.ClickException(
+        f"evaluation job {job_id} {status}: {job.get('error') or 'no reason recorded'}"
+    )
+
+
 @harbor.command("eval")
 @click.option(
     "--backend", "backend_id", required=True,
@@ -1018,15 +1123,12 @@ def evaluate_command(
         limits=EvaluationLimits(**limit_values) if limit_values else None,
         seed=seed,
     )
+    payload = body.model_dump(mode="json")
+    if detach:
+        click.echo(json.dumps(_request("POST", "/eval/jobs", payload=payload), indent=2))
+        return
     click.echo(
-        json.dumps(
-            _request(
-                "POST",
-                "/eval/jobs" if detach else "/eval",
-                payload=body.model_dump(mode="json"),
-            ),
-            indent=2,
-        )
+        json.dumps(_await_evaluation_job(_request("POST", "/eval/jobs", payload=payload)), indent=2)
     )