Repository navigation
Harbor design page and contributor checklists; scrub a CLI's live stream, refuse multi-step tasks as unsupported - #119
Merged
Conversation
…refactor After #117 the contributor checklists and the task tutorial still described the framework before it. Add docs/developer/harbor-design.md with three diagrams (the processes, a trial in order, CLI and loop agents) and who does each step, how the agent phase ends and the outcomes; let the sandbox checklist accept hermit.environment and count hermit/environment/ as core; add a benchmark checklist (task directories, public and private halves, what a task asks, parity, a real run) and say when it applies; point the task tutorial and the developer index to task directories; the dual-review reviewer is gpt-6.2-sol. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
19 of 21 tasks
…rub a CLI's live stream; refuse multi-step tasks as unsupported The independent review (/dual-review, 9 rounds) checked every claim of the Harbor design page, its diagrams and the checklists against the code and Harbor 0.25.0, and corrected what was false: Local is not confinement and sandbox evidence must show the isolation probe passed; only the bubblewrap task sandbox refuses a task the run cannot give, and every backend states its effective timeouts; Harbor's trial order, what is infra, which retries apply, what passes through the bridge, CLI install and reuse, the separate verifier. Two driver fixes the maintainer asked to include: - A CLI's output kept in /logs/agent as it ran reached a shared verifier unscrubbed when Harbor's timeout ended the phase before Hermit's logs were copied. The phase's end now scrubs each stream there, best effort, in a local Docker environment (never a failure of the phase's end). - A multi-step task was refused as sandbox_failed and run again; it is now refused as unsupported_task and never run again. Each has a test that fails on main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change
After the task-environment refactor (#117), the contributor checklists and the task tutorial still described the
framework before it, and the Harbor design was only in the long prose of
docs/framework/task.md#benchmarks.docs/developer/harbor-design.md("How Harbor runs a task"): the processes (Hermit, the driverunder Harbor's python, the task's container); who does each step of a trial; a trial in order with a command's path
through the bridge; CLI agents versus loop agents (three inline SVG diagrams,
docs/_includes/harbor-*.html, styledby
docs/_static/css/harbor-diagrams.cssfor both themes, as the sandbox diagrams are); how the agent phase ends; outcomes and retries; why Harbor'sTrial.run()owns the trial. Linked fromdocs/index.md,docs/developer/index.mdandtask.md#benchmarks,which stays the contract.
hermit.environment:Local,Sandboxed,Container, a task's Harbor environment), not only throughhermit.sandbox/hermit/isolate.py;a Harbor task's trial directory counts as its launch record;
hermit/environment/joins the files whose change isa core change.
.agents/skills/prepare-pr/references/benchmark-checklist.md): task directories inHarbor's format, the public and private halves, what each task asks of its environment, parity for a moved
benchmark, a real run.
prepare-pr/SKILL.mdandCONTRIBUTING.mdsay when it applies.docs/tutorials/your-own-task.md) and the developer index's "New task" row point to taskdirectories and Benchmarks.
.agents/skills/dual-review: the independent reviewer is Codexgpt-6.2-sol(wasgpt-6.1-sol).hermit/environment/harbor_driver.py, found in the review, included at the maintainer'srequest; CHANGELOG under Fixed):
/logs/agentas it ran reached a shared verifier unscrubbed when Harbor's agent timeoutended the phase before Hermit's own logs were copied over it. The phase's end now scrubs each such stream of the
credentials passed to the agent's commands, in a local Docker environment (whose
/logs/agentis the trial'sfolder on this machine): read without following a link, scrubbed before any copy is written, put back through
safe_copyonly when changed. It is best effort: a stream that cannot be read or replaced is logged and left asit came (as on
main), never a failure of the phase's end. Elsewhere a shared verifier still gets it as it came.[[steps]]) was refused assandbox_failedand run again afterinfra_wait_s; it is nowrefused as
unsupported_taskand never run again.main(Remove the forwards of moved modules #121) into the branch; no conflict.Validation
bash docs/build.sh(Sphinx, warnings as errors): passes; the three diagrams checked in light and dark (screenshots).hermit/environment/harbor.py,harbor_driver.py(the JSON-linesprotocol,
setsidand process groups, one transfer at a time, the end of the phase: any failure there is infra),harbor_errors.pyandtask.md#benchmarks(the 60 s margin,setup_timeout_sec1200 s,infra_retries2 after300 s,
busy_retries3, unscored runs).python -m pytest --changed --base origin/main(every group:hermit/environment/is shared) on a snapshot of thefinal tree: 1941 passed, 118 skipped, 0 failed (23 min).
mainand passing here:test_a_clis_live_stream_is_scrubbed_before_the_verifier_even_when_harbors_timeout_ends_the_phase(the stream andits key registered through
Driver.exec, Harbor's timeout without Hermit's logs, a planted link never followed, anon-mounted environment left alone, an unreadable stream never infra) and
test_a_multi_step_harbor_task_is_refused_as_unsupported_and_never_run_again. Both run under Harbor's venv, likethe other driver tests (skipped without it).
tests/core/test_harbor_environment.py: 115 passed;ruff check hermit/environment/harbor_driver.py: clean.Review
are the code's defaults (above).
hermit/environment/harbor_driver.py(the two fixes above), a sandbox change by the sandboxchecklist; agreed with the repository's maintainer, who asked for them. The checklists change what later PRs
answer.
the stream left as on
main); a link is never followed; a partial staging write is never published; themulti-step refusal maps to
unsupported_task, which is not retried.execand_end_phaseunder Harbor's python; the docs build. Not run: a live Docker trial that times out with a key in the stream./dual-review, this session and Codexgpt-6.2-sol, 9 rounds.Findings per round (Codex): 15, 13, 9, 6, 6, 5, 8, 3, 3; every one verified against the code. Fixed: the false
claims in the page, diagrams and checklists, the two driver fixes, and regressions the scrub briefly introduced
(made best effort). Left, under the maintainer's bar (fix only breakage, security, wrong results or cost, false
statements): NUL-delimited streams, a host umask of 077, a
mountsoverride of/logs/agent, a multi-step taskthat Modal compose rejects in
Trial.create, a stalled filesystem past the phase-end watchdog,safe_copyleaving a partial file on ENOSPC (predates this PR), wording and accessibility nits.
Test checklist: the new tests are in
tests/core/, named as sentences, with docstrings saying why; each fails onmain. Benchmark checklist: not applicable (no benchmark).🤖 Generated with Claude Code