Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ python -m pytest --changed # the test groups this branch touches (tests/READM
- Inspect the relevant implementation, callers and tests to understand current
behavior. Agents follow [Model → Base Agent → Acting / Evolver Agent](docs/framework/agent-layers.md):
reuse shared model and CLI execution through Base Agents; keep task-specific
logic in `examples/`. Where an agent runs is the environment its call gives (`hermit.agent.environment`), never
logic in `examples/`. Where an agent runs is the environment its call gives (`hermit.environment`), never
a setting; a benchmark's own agent subclasses a core agent (a `ReactAgent` with `@tool` methods and hooks) in its
example folder instead of copying a loop; a different prompt or model is configuration, not a new agent.
- For method reproductions, read the [reproduction guide](docs/developer/reproducing-a-method.md)
Expand Down
76 changes: 75 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,81 @@ Changes are grouped by software version. Entries under Unreleased describe work

### Breaking

- Harbor gives a task set its data, its environments and its verifier; every agent is Hermit's
([Benchmarks](docs/framework/task.md#benchmarks)). `agent.name: harbor` is gone, with `hermit/harbor/runner.py`,
`hermit/harbor/harborside/` (the HermitXXX classes, the scripted agent, the stream driver) and the runner's parts
(`clis`, `launch`, `staging`, `context`, `curate`, `errors`, `judge`, `signin`, `stream`, `tools`). A run config
names the task set's environments as `environment.harbor` (the settings that were `agent.params`: `benchmark`,
`tasks_dir`, `environment`, `verifier`, the multipliers, `agent_timeout_sec`, `docker_cleanup` and the disk gates,
`jobs_dir`, `harbor`, `pass_reward`, `reward_key`, `verifier_timeout`), its acting agent as any other
(`agent.base: codex` or `claude` with `agent.model` and `agent.<cli>.web_search`; `agent.name: answer` for the
answer-only agent that replaces HermitAnswer), and its judge as `{path: hermit.task.verify, name: environment}`.
Each task's environment is Harbor's own trial of it, run whole by Harbor's `Trial.run()` (`hermit/environment/
harbor.py` and its driver under Harbor's venv), whose agent is a bridge relaying the Hermit agent's commands
(`HermitBridge`: its setup installs what the agent declares, `Agent.installs`; its run ends its commands still
running before Harbor verifies): the run starts it behind the disk gate, hands it to the agent (`run(task, adapter,
environment=...)`), ends the agent phase and reads the trial's verdict; `hermit/harbor/docker.py` moved unchanged to
`hermit/environment/harbor_docker.py`. Codex and Claude Code run in it through its `exec` (isolation `remote`); the
agent has no worker (its loop runs in the run's process, every command in the task's container; the adapter's own
code, its context tool and its exports' render, in the strict adapter sandbox as before), and a CLI there signs in
by its model binding or a named `login` only (no copy of this machine's own sign-in). A rollout
that is not the agent's result keeps the runner's semantics (`hermit/environment/harbor_errors.py`, ported from
`hermit/harbor/errors.py`): it runs again in a fresh environment (`environment.harbor.infra_retries`, default 2,
after `infra_wait_s`, never for `disk` or `model_rejected`), and one that stays infra is `unscored`, which the judge
refuses (a failed run, never the agent's 0). A session in a task's environment is not resumed (refused: its home
there is made anew for each call). Eval rows of such a task keep its `outcome` (the runner's status); a setting
of `environment.harbor` is overridden on the command line by its path. The CLI agent's usage now counts every
attempt it ran (a provider it ran again on, as the runner counted a trial's runs); a provider that could not serve
runs the task again in a fresh environment (`environment.harbor.busy_retries`, `busy_wait_s`, the runner's).
An acting agent's model a CLI reaches (`agent.params.model: {cli: ...}`) is refused for a task in Harbor's
environments (its plan's wait would fall inside the task's time). What the runner read of a
session (a lost sign-in or a bill refused: the run ends; a configuration refused: `model_rejected`; the plan's
refusal: run again once it allows) is read the same way (`hermit/agent/base/codex_stream.py`, ported). Codex there
keeps the runner's `project_doc_max_bytes` of 262144 unless the config sets one. A hook audit that cannot be read
back leaves the session's result and usage (`hooks_audit_missing`). A task's environment refuses Hermit's network,
hidden-path and mount limits (the task's own say what it reaches). As the runner did:
`verifier_redact`, the named `reward_key` only (none there is infra), the agent's setup (a CLI's install) within
1200 s unless `environment.harbor.setup_timeout_sec` says (Harbor's own default, 360 s, is too short under load), a ChatGPT sign-in taken back only when valid
(64 KiB, both tokens, its clock), a session's missing usage as an unknown cost. Not carried
over: multi-step tasks (streams), the adapter's tools in a task's environment, the `<cli>_bin` mount (a config
mounts a binary through `environment.harbor.environment.mounts` and names it as `agent.<cli>.bin`), Harbor's own agents
(`acting_agent: {name}`), the Pi, OpenCode, dsh and mini-swe-agent CLIs in a task's environment, and the curated
trial log (the trial's directory under `jobs_dir` is kept as Harbor writes it). Every example config moved.

- OfficeQA's task set is task directories in Harbor's format under `examples/officeqa/utils/benchmark/` (with its
`prepare.py`, `judge.py`, `task-info/`, splits, `data.json` and seed adapter): run `prepare.py` again. The agents read
each task as the row they always read, so prompts are unchanged.

- More task sets are task directories in Harbor's format, under each example's `utils/benchmark/` with its
`prepare.py`, `judge.py`, `task-info/` and splits; their configs read each task through `data.view` as the row they
always read, so prompts, records and learning material are unchanged (`hermit.task.directory.write_task` writes
one). `examples/claude_subscription/`: its nine tasks, judge, splits and task-info moved to `utils/benchmark/`.
`examples/aspire_robosuite/` and `examples/libero/`: their launchers run `utils/benchmark/prepare.py`;
`sim_service.py` and `stage_docs.py --tasks` take the task directories.
`examples/robodojo_sim/`: its launcher runs `utils/benchmark/prepare.py`; `dojo_service.py --tasks` takes the task
directories.
`examples/servelearnbench/`: run `utils/benchmark/prepare.py` again (`windows.py` and `windows.json` moved there
too).
`examples/clbench_database/` and `examples/clbench_poker/`: run `utils/benchmark/prepare.py` again (`task-info/`
moved there too).
`examples/agentdyn/` and `examples/agentdojo/`: prepare each protocol again under `utils/benchmark/data/` (each
writes `tasks/`, `cases.txt` with every case in its order, the splits and the manifest, whose `tasks_sha256` is still
that of the rows as JSON lines, as `DATA.md` lists them); `report.py` and `compare.py --tasks` take a protocol's
`tasks/`.

- Tasks and environments are packages of their own beside the agents: `hermit/environment/` (moved from
`hermit/agent/environment/`, which forwards for one release) and `hermit/task/` (from `hermit/task.py`: `Task` with
a public half the agent is handed and a private half only the judge reads, `Benchmark`, the verifier). `RunOutput`
and `load_callable` live in `hermit/common.py`; every name importable from `hermit.task` before still is. Tasks and
agents never import each other (`tests/core/test_layers.py`).

- One agent abstraction (#111, [agent layers](docs/framework/agent-layers.md#from-the-one-agent-refactor)): where an
agent works is an environment given with each call. `BaseAgent.run(prompt, environment=..., tools=, session=,
roles=)` replaces `run(prompt, workspace=, limits=)` (a path is a `Local` environment), and a CLI Base Agent
refuses `isolation`, `sandbox` or `container` given as settings (use `Sandboxed(path, sandbox=...)` or
`Container(path, settings=...)`). A single model call is Layer 1: `Direct(settings, model)` in
`hermit.agent.model.direct` returns a `Reply`, and `make("direct")` is refused. `hermit.agent.base.direct` and
`hermit.agent.base.container` (now `hermit.agent.environment.docker`) forward for one release. A run whose acting
`hermit.agent.base.container` (now `hermit.environment.docker`) forward for one release. A run whose acting
agent does not take what its method needs of an adapter (`supports` against `evolver.yaml` `needs`) is refused
before it starts.

Expand Down Expand Up @@ -47,6 +115,12 @@ Changes are grouped by software version. Entries under Unreleased describe work

### Added

- Task directories in Harbor's format as a task set (`data.tasks: <directory>`, `data.rollouts`), read without Harbor
(`hermit/task/directory.py`), with their own verifier script run in the judge's sandbox (`judge: {path:
hermit.task.verify, name: script}`: `/tests`, `/logs/verifier/reward.txt` or `reward.json`, and `feedback.txt`).
The task sandbox is built from an `EnvironmentSpec` by the bubblewrap backend (`hermit/environment/spec.py`,
`bwrap.py`), which refuses a task that asks for more network or memory than the run's environment gives.

- RoboDojo-Sim (`examples/robodojo_sim/`), a first implementation (its smoke run on the real simulator; no
training run yet): its forty-two tasks with a Codex session writing a program that controls the robot,
evaluated by RoboDojo as an XPolicyLab policy whose server runs the program in a sandbox with no network and no
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -300,7 +300,7 @@ Prepare the dataset and model endpoint as described in the [OfficeQA recipe](htt

```bash
python -m pip install -e . --group skillopt --group gepa
python examples/officeqa/utils/prepare.py --csv /path/to/officeqa_full.csv --docs /path/to/treasury_bulletins_parsed
python examples/officeqa/utils/benchmark/prepare.py --csv /path/to/officeqa_full.csv --docs /path/to/treasury_bulletins_parsed
hermit check --config examples/officeqa/gepa/smoke.yaml
hermit train --config examples/officeqa/gepa/smoke.yaml
hermit eval --config examples/officeqa/gepa/smoke.yaml --adapter final
Expand Down
6 changes: 3 additions & 3 deletions docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,13 +69,13 @@ before the offline runs; running it again re-syncs the clbench venv, so not whil
hermit runs every agent, learner and adapter tool in a bubblewrap sandbox of its own (`hermit/sandbox.py`): user,
mount, PID and network namespaces, a fresh `/proc` and `/dev`, a positive allowlist of paths. Docker allows an
unprivileged process to make namespaces only with `--cap-add SYS_ADMIN`, `--security-opt seccomp=unconfined` and
`--security-opt apparmor=unconfined`, the list of `hermit/agent/environment/docker.py`, which runs Codex's bubblewrap the same way. One
`--security-opt apparmor=unconfined`, the list of `hermit/environment/docker.py`, which runs Codex's bubblewrap the same way. One
more is needed for hermit's `--proc /proc`: `--security-opt systempaths=unconfined` lifts the masks Docker lays over
`/proc` (`/proc/kcore`, `/proc/keys`, ...), without which a fresh `/proc` cannot be mounted in a namespace
(`bwrap: Can't mount proc on /newroot/proc: Operation not permitted`); `docker.py` gives it to Claude Code's
containers too, whose own sandbox mounts one. The container's user is unprivileged, so what
the masks hid stays root's (`/proc/kcore` is mode 0400, `/proc/sys` is read-only to it). `NET_RAW`, `MKNOD` and
`AUDIT_WRITE` are dropped as `hermit/agent/environment/docker.py` drops them, and `label=disable` keeps an SELinux host from refusing
`AUDIT_WRITE` are dropped as `hermit/environment/docker.py` drops them, and `label=disable` keeps an SELinux host from refusing
the bind mounts. The containers are not privileged, have no Docker socket and are not on the host network.
`prepare` gets none of the sandbox additions.

Expand All @@ -85,7 +85,7 @@ the bind mounts. The containers are not privileged, have no Docker socket and ar
|---|---|---|
| `HERMIT_DIR` | read-write | `PYTHONPATH`; the examples' `utils/` (tasks, splits, `data.json` with absolute paths, the `documents` link) and `runs/`. |
| `CLBENCH_DIR` | read-write | `.venv/bin/python` is a link to the interpreter under `.uv-python/`, and `pyvenv.cfg` names it by absolute path; hermit's task sandbox grants `$MWM_CLBENCH` and binds the interpreter the venv names. `prepare` puts the interpreter, the venv and uv's cache under the checkout (`UV_PYTHON_INSTALL_DIR`, `UV_CACHE_DIR`) so they resolve on the host too. A venv made on the host with an interpreter elsewhere does not resolve inside. |
| `OFFICEQA_DIR` | read-only | `examples/officeqa/utils/documents` links to it and `data.json` names it. |
| `OFFICEQA_DIR` | read-only | `examples/officeqa/utils/benchmark/documents` links to it and `data.json` names it. |
| `RUN_HOME` | read-write | `HOME` inside: `~/.cache` (uv, Hugging Face, the tests' `hermit-tests`), a CLI agent's home; the launchers' `$HOME/code/...` defaults land here unless overridden. |
| `/tmp` | tmpfs | A fresh, memory-backed `/tmp` per container (`exec`, 8 GB); hermit's sandbox bridges live there. |

Expand Down
6 changes: 3 additions & 3 deletions docker/offline.sh
Original file line number Diff line number Diff line change
Expand Up @@ -113,11 +113,11 @@ echo "== clbench: the database venue'\''s products.db"
"$CLBENCH_DIR/.venv/bin/python" -c "from src.tasks.database_exploration.task import DatabaseExploration as T; T.setup()"
cd "$HERMIT_DIR"
echo "== examples/clbench_database"
python examples/clbench_database/utils/prepare.py --clbench "$CLBENCH_DIR"
python examples/clbench_database/utils/benchmark/prepare.py --clbench "$CLBENCH_DIR"
echo "== examples/clbench_poker"
python examples/clbench_poker/utils/prepare.py --clbench "$CLBENCH_DIR"
python examples/clbench_poker/utils/benchmark/prepare.py --clbench "$CLBENCH_DIR"
echo "== examples/officeqa"
python examples/officeqa/utils/prepare.py --csv "$OFFICEQA_DIR/officeqa_full.csv" --docs "$OFFICEQA_DIR/treasury_bulletins_parsed"
python examples/officeqa/utils/benchmark/prepare.py --csv "$OFFICEQA_DIR/officeqa_full.csv" --docs "$OFFICEQA_DIR/treasury_bulletins_parsed"
'

ENVFILE=
Expand Down
4 changes: 3 additions & 1 deletion docs/developer/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,11 @@ Start with the extension boundary you need. A task integration normally requires
|---|---|
| `hermit/cli.py`, `config.py` | Command parsing and config resolution |
| `hermit/agent/` | The [agent layers](../framework/agent-layers.md): `model/` (Layer 1), `base/` (Layer 2, the Base Agents), `acting/` and `evolver/` (Layer 3) |
| `hermit/environment/` | Where an agent or a judge works: the `Environment` interface and its backends (local, bwrap sandbox, Docker) |
| `hermit/agent/acting/` | The Acting Agent interface and the built-in acting agents |
| `hermit/agent/evolver/` | The Evolver interface and the agent Evolver; `methods/` holds the built-in learning methods |
| `hermit/task.py` | Task records, callable loading, pipeline/judge types |
| `hermit/task/` | Tasks: `Task` (public and private halves), `Benchmark` (a task set and its splits), the judge's verdict, the records a run keeps |
| `hermit/common.py` | What every layer shares: `RunOutput` (an acting agent's result) and callable loading |
| `hermit/adapter/` | Manifest loading and acting consumption |
| `hermit/adapter/templates/` | Built-in adapter seeds and specifications |
| `hermit/trainer/` | Training loop, rollouts, material, budgets, recovery |
Expand Down
Loading
Loading