Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .agents/skills/dual-review/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: dual-review
description: Review a PR with two independent reviewers (this session and a fresh Codex gpt-6.1-sol at high effort), merge and verify their findings, fix, and repeat until the PR is ready for merge; minor points are recorded, not fixed. Ends with the merge verdict, the changes made and every unresolved finding with its reason. Invoke with /dual-review <PR number | URL | branch> [repo].
description: Review a PR with two independent reviewers (this session and a fresh Codex gpt-6.2-sol at high effort), merge and verify their findings, fix, and repeat until the PR is ready for merge; minor points are recorded, not fixed. Ends with the merge verdict, the changes made and every unresolved finding with its reason. Invoke with /dual-review <PR number | URL | branch> [repo].
user-invocable: true
---

Expand All @@ -14,7 +14,7 @@ left alone, since fixing them costs rounds and tends to break real paths. It sto
that blocks the merge.

Run it in a Claude Opus 5.5 session at high effort (`/model`, `/effort`); say so at the start if the session is on
something else. The independent reviewer is always Codex with `gpt-6.1-sol` at `model_reasoning_effort="high"`.
something else. The independent reviewer is always Codex with `gpt-6.2-sol` at `model_reasoning_effort="high"`.

## Inputs

Expand Down Expand Up @@ -45,7 +45,7 @@ status) so it does not re-report them. Then run, with `run_in_background: true`
`W` the worktree):

```bash
codex exec --skip-git-repo-check -C "$W" -s read-only -m gpt-6.1-sol -c model_reasoning_effort="high" \
codex exec --skip-git-repo-check -C "$W" -s read-only -m gpt-6.2-sol -c model_reasoning_effort="high" \
-o "$S/dual-review-<pr>-r<N>-codex.md" - < "$S/dual-review-<pr>-r<N>-prompt.md" \
> "$S/dual-review-<pr>-r<N>-codex.log" 2>&1
```
Expand Down
2 changes: 2 additions & 0 deletions .agents/skills/prepare-pr/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ affected interfaces, and the validation evidence available.
[test checklist](references/test-checklist.md).
- For PRs that add or change a method or a dataset and start a process or change
what a sandbox shows, also complete the [sandbox checklist](references/sandbox-checklist.md).
- For PRs that add a benchmark or change how a benchmark's tasks are written or read,
also complete the [benchmark checklist](references/benchmark-checklist.md).

Keep checklist questions in their owning files. Answer shared items once, using
actual code, checks and records rather than planned work.
Expand Down
28 changes: 28 additions & 0 deletions .agents/skills/prepare-pr/references/benchmark-checklist.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Benchmark checklist

Answer these items alongside the [general checklist](general-pr-checklist.md) when a pull request adds a benchmark
or changes how a benchmark's tasks are written or read. The contract is in
[Task directories](../../../../docs/framework/task.md#task-directories) and
[Benchmarks](../../../../docs/framework/task.md#benchmarks); how a task runs in Harbor is in
[How Harbor runs a task](../../../../docs/developer/harbor-design.md).

- [ ] **Task directories.** Are the tasks task directories in Harbor's format, rather than a new row format?
`hermit.task.directory.write_task` is the standard helper for a `prepare.py`; `python -m hermit.harbor.dataset
prepare <benchmark.yaml>` prepares the Harbor dataset its `dataset.ref` names (give the reason for each
`dataset.patch` configured). A task set whose tasks each
need an image of their own runs in Harbor's environments with `hermit.task.verify`, never a runner of its own.
- [ ] **Public and private halves.** Do the rows the agent and the learner read hold only the public half (the
instruction, its settings, its metadata), with answers, tests and the verifier in the private half or in
judge-owned storage? Name each field a `data.view` adds and why the learner may see it, and what private
information the judge's feedback may show under the benchmark's protocol.
- [ ] **What a task asks.** Does each task's `task.toml` declare the network (`network_mode`, `allowed_hosts`),
memory and timeouts its agent and its verifier need? Hermit's bubblewrap task sandbox refuses a task whose network
or memory the run cannot give; on the host, with `environment.image`, or in Harbor's environments (where the run's
overrides apply: `environment.harbor.environment.override_memory_mb`, ...), state how the configs meet them. On
every backend, state the effective timeouts: a run's own (`agent.timeout_min`,
`environment.harbor.agent_timeout_sec`) may be shorter than a task's.
- [ ] **Parity, for a moved benchmark.** For a benchmark moved from rows to task directories, do its tasks read back
through each of its configs as exactly the old rows (prompts, learning material, splits)? State how this was
compared, with the counts.
- [ ] **A real run.** Link one smoke run through `hermit eval` (and `hermit train` when a method is part of the
change), with its scores and no infrastructure errors.
20 changes: 13 additions & 7 deletions .agents/skills/prepare-pr/references/sandbox-checklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,17 @@ changes a method or a dataset starts a process or changes what a sandbox shows.
[How the sandbox works](../../../../docs/guides/sandbox.md).

- [ ] **Reuse the framework's sandbox.** Does every process the change starts (agent, judge, learner and its roles,
tools, the adapter's code) run through `hermit.sandbox` or `hermit/isolate.py`, with its needs declared in the
existing settings (`environment.*`, `judge.*`, the method's view, `sandbox.evolver.*`)? Link one
`sandbox/<role>-<n>.json` from a run of the change. For each process that cannot use this abstraction, state why
and how it is confined instead.
tools, the adapter's code) run in a confined environment the framework gives (`hermit.environment`: `Sandboxed`,
`Container`, or a task's Harbor environment) or through `hermit.sandbox` / `hermit/isolate.py`, with its needs
declared in the existing settings (`environment.*`, `judge.*`, a task's `task.toml`, the method's view,
`sandbox.evolver.*`)? `Local` is not confinement: it runs on this machine as it is. Link the evidence from a run of
the change that the confinement was enforced: a launch record (`sandbox/<role>-<n>.json` whose isolation probe
passed, `probe.passed: true`), or, where the framework keeps no launch record (a container, a Harbor trial with its
`trial_dir` in the rollout's verdict, the strict adapter sandbox), a check run inside it that shows what it can read,
write and reach. For each process that runs otherwise (`Local`
included), state why and how it is confined instead, or, for trusted code on the host (a judge that only reads a
verdict), exactly what it does.
- [ ] **Sandbox changes are core changes.** Does the pull request change the sandbox itself (`hermit/sandbox.py`,
`hermit/sandbox_relay.py`, `hermit/isolate.py`, the Trainer's evolver sandbox, or their defaults)? If so, link the
issue where the change was agreed with the core team, and answer it under **Core changes** in the general
checklist.
`hermit/sandbox_relay.py`, `hermit/isolate.py`, `hermit/environment/`, the Trainer's evolver sandbox, or their
defaults)? If so, link the issue where the change was agreed with the core team, and answer it under **Core
changes** in the general checklist.
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -423,6 +423,14 @@ Changes are grouped by software version. Entries under Unreleased describe work

### Fixed

- In a task's Harbor environment, a CLI's output kept in `/logs/agent` as it ran reached a shared verifier unscrubbed
when Harbor's agent timeout ended the phase before Hermit's own logs were copied over it. The phase's end now scrubs
each such stream there of the credentials passed to the agent's commands (their variables and descriptors) before
the verifier runs, in a local Docker environment, whose `/logs/agent` is the trial's folder on this machine (best
effort: a stream this machine cannot read stays as it came). Elsewhere a shared verifier still gets it as it came;
the trial's files are scrubbed after the trial.
A multi-step task (`[[steps]]`) was refused as an environment that failed to start and run again after
`infra_wait_s`; it is now refused as unsupported (`unsupported_task`) and never run again.
- `Sandbox.wrap` ran a virtual environment's interpreter (`<venv>/bin/python`, a link to its base interpreter) by
its real path, so it started without the environment and its packages failed to import. An interpreter whose
environment is granted (the sandbox's `readable` or `writable`, the call's, or `interpreters`) now runs by its own
Expand Down
5 changes: 5 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,11 @@ also answers the [sandbox checklist](.agents/skills/prepare-pr/references/sandbo
framework's sandbox, states why for anything that cannot, and treats a change to the sandbox itself as a core
change agreed with the core team first.

A pull request that adds a benchmark or changes how its tasks are written or read also answers the
[benchmark checklist](.agents/skills/prepare-pr/references/benchmark-checklist.md): task directories in Harbor's
format, answers and tests kept out of the tasks the agent and the learner read (the judge's feedback shows what the
benchmark's protocol allows), what each task asks of its environment declared, and parity for a moved benchmark.

GitHub Actions (`.github/workflows/tests.yml`) runs `python -m pytest --changed` against the base branch on every
pull request, and the complete suite on main, daily and on demand, on Ubuntu with bubblewrap. Tests whose
dependencies it does not install (agent CLIs, AgentDyn, prepared benchmark data) skip there, as they do locally.
Expand Down
45 changes: 45 additions & 0 deletions docs/_includes/harbor-agents.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
<figure class="ha-hb">
<div class="ha-hb-scroll">
<svg viewBox="0 0 920 300" role="img" aria-label="A CLI agent passes through the bridge only to set up and start a session and to bring back its files; the CLI runs every command in the container itself and calls the model from there; Hermit downloads what it left. A loop agent's loop and model calls stay in Hermit, and each of its commands is one exec through the bridge">
<defs>
<marker id="hba-m" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="ah-hermit" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="hba-p" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="ah-plain" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<text class="t-h" x="20" y="22">CLI agent: Codex, Claude Code</text>
<rect class="b-hermit" x="20" y="36" width="170" height="96" rx="8"/>
<text class="t-b" x="105" y="62" text-anchor="middle">Hermit</text>
<text class="t-m" x="105" y="84" text-anchor="middle">renders its home, prompt</text>
<text class="t-m" x="105" y="102" text-anchor="middle">reads its stream, logs</text>
<line class="a a-hermit" x1="190" y1="70" x2="316" y2="70" marker-end="url(#hba-m)"/>
<text class="t-m" x="253" y="62" text-anchor="middle">set up, then run</text>
<line class="a a-hermit" x1="316" y1="104" x2="194" y2="104" marker-end="url(#hba-m)"/>
<text class="t-m" x="253" y="122" text-anchor="middle">its files at the end</text>
<rect class="b-box" x="320" y="36" width="410" height="96" rx="8"/>
<text class="t-b" x="336" y="58">task's container</text>
<rect class="b-hermit" x="336" y="68" width="120" height="52" rx="6"/>
<text x="396" y="98" text-anchor="middle">CLI session</text>
<line class="a a-plain" x1="456" y1="94" x2="502" y2="94" marker-end="url(#hba-p)"/>
<text class="mono" x="510" y="84">ls · git · pytest · edits</text>
<text class="t-m" x="510" y="104">run by the CLI itself</text>
<text class="t-m" x="510" y="122">none through the bridge</text>
<line class="a a-plain" x1="730" y1="70" x2="820" y2="70" marker-end="url(#hba-p)"/>
<rect class="b-inner" x="824" y="48" width="80" height="44" rx="6"/>
<text class="t-m" x="864" y="75" text-anchor="middle">model</text>
<text class="t-m" x="775" y="62" text-anchor="middle">calls</text>
<text class="t-h" x="20" y="182">Loop agent: a ReactAgent, the answer-only agent</text>
<rect class="b-hermit" x="20" y="196" width="170" height="96" rx="8"/>
<text class="t-b" x="105" y="222" text-anchor="middle">Hermit</text>
<text class="t-m" x="105" y="244" text-anchor="middle">the agent's loop is here</text>
<text class="t-m" x="105" y="262" text-anchor="middle">Hermit calls the model</text>
<line class="a a-hermit" x1="190" y1="226" x2="316" y2="226" marker-end="url(#hba-m)"/>
<line class="a a-hermit" x1="190" y1="244" x2="316" y2="244" marker-end="url(#hba-m)"/>
<line class="a a-hermit" x1="190" y1="262" x2="316" y2="262" marker-end="url(#hba-m)"/>
<text class="t-m" x="253" y="284" text-anchor="middle">1 exec per command</text>
<rect class="b-box" x="320" y="196" width="410" height="96" rx="8"/>
<text class="t-b" x="336" y="218">task's container</text>
<text class="t-m" x="336" y="246">runs the one command it is sent, returns the result</text>
<text class="t-m" x="336" y="266">once for each command the agent runs</text>
</svg>
</div>
<figcaption>A CLI is an ordinary process in the container, like a terminal opened there; the bridge only sets it up, starts it and brings back what it left. A loop agent's commands each pass through the bridge.</figcaption>
</figure>
57 changes: 57 additions & 0 deletions docs/_includes/harbor-processes.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
<figure class="ha-hb">
<div class="ha-hb-scroll">
<svg viewBox="0 0 920 330" role="img" aria-label="Three processes: Hermit runs the acting agent and its judge; a driver under Harbor's python runs Harbor's Trial.run with HermitBridge as its agent; the task's container runs the agent's commands and the verifier. Hermit and the bridge speak JSON lines; the bridge relays exec, upload and download into the container; Trial.run runs the verifier and hands its verdict to Hermit's judge">
<defs>
<marker id="hbp-m" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="ah-hermit" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="hbp-h" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="ah-harbor" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<!-- Hermit -->
<rect class="b-hermit" x="20" y="20" width="240" height="290" rx="10"/>
<text class="t-h c-hermit" x="36" y="44">Hermit process</text>
<rect class="b-inner" x="36" y="60" width="208" height="56" rx="6"/>
<text class="t-b" x="52" y="84">acting agent</text>
<text class="t-m" x="52" y="103">starts a CLI, or runs a loop</text>
<line class="a a-hermit" x1="140" y1="116" x2="140" y2="142" marker-end="url(#hbp-m)"/>
<rect class="b-inner" x="36" y="146" width="208" height="56" rx="6"/>
<text class="mono t-b" x="52" y="170">HarborEnvironment</text>
<text class="t-m" x="52" y="189">the Environment API</text>
<rect class="b-inner" x="36" y="236" width="208" height="56" rx="6"/>
<text class="t-b" x="52" y="260">judge</text>
<text class="mono t-m" x="52" y="279">hermit.task.verify</text>
<!-- driver -->
<rect class="b-harbor" x="370" y="20" width="240" height="290" rx="10"/>
<text class="t-h c-harbor" x="386" y="44">driver · Harbor's python</text>
<rect class="b-inner" x="386" y="60" width="208" height="232" rx="6"/>
<text class="mono t-b" x="402" y="84">Trial.run()</text>
<rect class="b-hermit" x="402" y="120" width="176" height="64" rx="6"/>
<text class="t-b" x="418" y="146">HermitBridge</text>
<text class="t-m" x="418" y="166">relays; no logic of its own</text>
<text class="t-m" x="402" y="214">pull or build · setup</text>
<text class="t-m" x="402" y="232">agent phase · verifier</text>
<text class="t-m" x="402" y="250">stop · result.json · scrub</text>
<!-- container -->
<rect class="b-box" x="710" y="20" width="190" height="290" rx="10"/>
<text class="t-h" x="726" y="44">task's container</text>
<rect class="b-inner" x="726" y="60" width="158" height="124" rx="6"/>
<text class="t-b" x="740" y="84">agent's commands</text>
<text class="t-m" x="740" y="106">a loop agent's commands,</text>
<text class="t-m" x="740" y="124">or one CLI session</text>
<text class="t-m" x="740" y="142">running its own</text>
<rect class="b-inner" x="726" y="236" width="158" height="56" rx="6"/>
<text class="mono t-b" x="740" y="260">tests/test.sh</text>
<text class="t-m" x="740" y="279">only for the verifier</text>
<!-- links -->
<line class="a a-hermit" x1="248" y1="162" x2="398" y2="162" marker-start="url(#hbp-m)" marker-end="url(#hbp-m)"/>
<text class="mono" x="315" y="154" text-anchor="middle">JSON lines</text>
<text class="t-m" x="315" y="180" text-anchor="middle">stdin, stdout</text>
<line class="a a-harbor" x1="582" y1="152" x2="722" y2="152" marker-start="url(#hbp-h)" marker-end="url(#hbp-h)"/>
<text class="t-m" x="660" y="144" text-anchor="middle">exec · upload</text>
<text class="t-m" x="660" y="170" text-anchor="middle">download</text>
<line class="a a-harbor" x1="594" y1="264" x2="722" y2="264" marker-end="url(#hbp-h)"/>
<text class="t-m" x="660" y="256" text-anchor="middle">verifier</text>
<line class="a a-harbor" x1="386" y1="270" x2="248" y2="270" marker-end="url(#hbp-h)"/>
<text class="t-m" x="315" y="262" text-anchor="middle">its verdict</text>
</svg>
</div>
<figcaption>Purple is Hermit's, teal is Harbor's. The bridge is the agent Harbor's trial runs; it carries Hermit's agent's commands into the container and decides nothing itself. The verifier is shown in shared mode, in the agent's container.</figcaption>
</figure>
Loading
Loading