Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,38 @@

### Changed

- The campaign chart renderer and the single-task launcher move into the Apps
package and the two root modules are removed (Refs #408, A2), with no
forwarding shell, `sys.modules` alias or import fallback. Both files move
with their bytes unchanged:
- `inferrsi.summary_chart` → `inferrsi.apps.views.summary_chart`
(`src/inferrsi/summary_chart.py` →
`src/inferrsi/apps/views/summary_chart.py`), which keeps `build_svg`,
`build_markdown`, `spec_from_evaluation`, `check_trial_chart_eligibility`,
`SpecError` and `main`
- `inferrsi.benchmark_operations` → `inferrsi.apps.benchmark_operations`
(`src/inferrsi/benchmark_operations.py` →
`src/inferrsi/apps/benchmark_operations.py`), which keeps `run_task`
- `tools/campaign_summary_chart.py` keeps its command line and arguments.
Its exit codes, stdout, stderr and the SVG and Markdown bytes it writes are
unchanged. The dashboard, the HTML report and `benchmarks run` import the
canonical modules.
- Importing `inferrsi.summary_chart` or `inferrsi.benchmark_operations` now
raises `ModuleNotFoundError`, in external Python code as well as in this
repository, and a pickle that names an object by either path no longer
loads. Callers import from `inferrsi.apps.views.summary_chart` and
`inferrsi.apps.benchmark_operations`. Frozen evidence is not rewritten.
- The tracked package root shrinks from eight files to six. Besides
`__init__.py` and `__main__.py`, the four left (`mindie_context.py`,
`mindie_mcp.py`, `mindie_project.py` and `_mindie_read_worker.py`) wait for
M1 (#452). The gate's exact-file `APPLICATION_ENTRIES` listing is removed
with its last entry, so no root file is application assembly, and
`inferrsi.summary_chart` and `inferrsi.benchmark_operations` join
`RETIRED_ROOT_MODULES`, so a root file that returns under either name
fails.
- Isolation is still incomplete: the 12 Harness hits into Infra (R2, #412)
and the five runtime resource loaders (R3, #412) remain.

- The CLI moves into the Apps package and the two root CLI modules are removed
(Refs #408, A1), with no forwarding shell, `sys.modules` alias or import
fallback:
Expand Down
20 changes: 11 additions & 9 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ packages. `inferrsi benchmarks run <package> ...` launches the package's
`benchmark.py`. Task-specific orchestration, sampling and assessment live inside
each package. The task library owns its `index.json` registration. The existing
`benchmark/registry.py` discovers the resources. The CLI package `apps/cli/`
and the existing `benchmark_operations.py` provide application assembly and launch.
and `apps/benchmark_operations.py` provide application assembly and launch.
Each task retains its own protocol and task-specific code.

The [current-Agent optimization session](optimization-benchmarks.md) assembles
Expand Down Expand Up @@ -159,7 +159,6 @@ the comparator; the engine AB/BA path retains its independent acceptance contrac
```text
src/inferrsi/
apps/cli/ CLI entry `main` and its argument parser
benchmark_operations.py Resolve a task entry and launch it through Harness
benchmark/
registry.py Generic task resource discovery
tasks/ Task-specific definitions, workloads and assessment
Expand All @@ -171,6 +170,7 @@ src/inferrsi/
improvement.py Improvement-step thresholds and acceptance rules
apps/
api.py Python entry points for assembling and publishing Apps operations
benchmark_operations.py Resolve a task entry and launch it through Harness
archive.py Publish and read the standardized Apps output in a workspace
result.py Assemble and check the AppsResultV1 output envelope
skill_evaluation.py Composed bundle export, trial run and summary
Expand All @@ -189,11 +189,11 @@ src/inferrsi/
resources/
__init__.py Import-free initializer
repository_packs.py Neutral REPO_TYPES and pack.json reader; imports no inferrsi module
summary_chart.py Stdlib SVG gain chart (chain/branch) and report figure spec
apps/
evidence_layout.py Create the plan/results/patch/validator evidence directories
campaign.py Knowledge values connected to Harness campaign records
views/
summary_chart.py Stdlib SVG gain chart (chain/branch) and report figure spec
optimization_chart.py Managed optimization objective/token curves and final result
dashboard.py Offline snapshot and HTML rendering
dashboard_data.py Recorded run/campaign evidence into a view model
Expand Down Expand Up @@ -271,17 +271,19 @@ examples/ Current Qwen runner, image validator and v4 adapter
tests/fixtures/legacy/ Frozen historical validation dependencies for regressions
```

The package root holds exactly eight tracked files at this stage: `__init__.py`,
`__main__.py`, `benchmark_operations.py`, `summary_chart.py`,
`mindie_context.py`, `mindie_mcp.py`, `mindie_project.py` and
`_mindie_read_worker.py`. #408 A1 moved the CLI and its parser to `apps/cli/`
and deleted root `cli.py` and `arguments.py` with no forwarding shell or alias;
The package root holds exactly six tracked files at this stage: `__init__.py`,
`__main__.py`, `mindie_context.py`, `mindie_mcp.py`, `mindie_project.py` and
`_mindie_read_worker.py`. #408 A2 moved the chart renderer to
`apps/views/summary_chart.py` and the task launcher to `apps/benchmark_operations.py`
and deleted root `summary_chart.py` and `benchmark_operations.py` with no forwarding
shell or alias. #408 A1 moved the CLI and its parser to `apps/cli/`
and deleted root `cli.py` and `arguments.py` the same way;
`__main__.py` and the console script call `inferrsi.apps.cli:main`. #412 R1 deleted eleven root modules with no
forwarding shell or alias: the seven Knowledge shells, whose code is in
`knowledge/`; the three old-run forwarders `evaluation.py`, `execution.py` and
`reporting.py`, whose code is in `apps/legacy/`; and `inspection.py`, whose
host binding is `apps/inspection.py`. The final two-file root (`__init__.py`
and `__main__.py`) waits for the A2 (#408) and M1 (#452) moves.
and `__main__.py`) waits for the M1 (#452) move.
[module-layout.md](module-layout.md#retired-root-modules) maps each retired
path to its replacement and lists what still keeps the isolation gate false.

Expand Down
2 changes: 1 addition & 1 deletion docs/campaign-summary-chart.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ One figure per campaign that answers "what did this campaign move, and by
how much". It is drawn automatically when an experiment finishes, and the same
renderer produces a standalone figure for a multi-step campaign.

The renderer is [inferrsi/summary_chart.py](../src/inferrsi/summary_chart.py),
The renderer is [inferrsi/apps/views/summary_chart.py](../src/inferrsi/apps/views/summary_chart.py),
also exposed as [tools/campaign_summary_chart.py](../tools/campaign_summary_chart.py)
and listed in the [operation guide](operations.md#summarize-a-campaign-figure--campaign_summary_chartpy).
It uses only the standard library and writes SVG, so a committed figure
Expand Down
6 changes: 3 additions & 3 deletions docs/engineering-benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,9 +57,9 @@ and TTS entries also run directly by file path.

```text
src/inferrsi/
arguments.py Existing CLI argument assembly
cli.py Existing command dispatch
benchmark_operations.py Application child launcher and executor injection
apps/
cli/ CLI command dispatch and argument assembly
benchmark_operations.py Application child launcher and executor injection
harness/
optimization.py Current-Agent session budget, records and recovery
usage.py Shared usage normalization and totals
Expand Down
55 changes: 34 additions & 21 deletions docs/module-layout.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ Knowledge move (K01) is implemented: `src/inferrsi/knowledge/` holds the seven
moved Knowledge bodies. R1 (#412) deleted the seven static root compatibility
shells that K01 left, so those names import only from `inferrsi.knowledge`.
The engineering task contribution now implements `benchmark/registry.py`
and `benchmark/tasks/`. The CLI package `apps/cli/` and the flat
`benchmark_operations.py` assemble this slice. The former `observability/`
and `benchmark/tasks/`. The CLI package `apps/cli/` and
`apps/benchmark_operations.py` assemble this slice. The former `observability/`
package is dissolved. Its
campaign records, traces, logs and profile captures are in `harness/records/`,
its dashboard views and templates are in `apps/views/`, and the
Expand Down Expand Up @@ -43,18 +43,20 @@ no forwarding shell or alias.

The flat package root is retired in stages.
`tests/test_package_root_inventory.py` pins the root to an exact file set: 23
tracked files after C1, 21 after C2, 10 after R1 (#412) and 8 after A1 (#408).
The eight are `__init__.py`, `__main__.py`, `_mindie_read_worker.py`,
`benchmark_operations.py`, `mindie_context.py`, `mindie_mcp.py`,
`mindie_project.py` and `summary_chart.py`. A1 moved root `cli.py` and
tracked files after C1, 21 after C2, 10 after R1 (#412), 8 after A1 (#408) and 6
after A2 (#408). The six are `__init__.py`, `__main__.py`,
`_mindie_read_worker.py`, `mindie_context.py`, `mindie_mcp.py` and
`mindie_project.py`. A2 moved root `summary_chart.py` to
`apps/views/summary_chart.py` and root `benchmark_operations.py` to
`apps/benchmark_operations.py` and deleted both root files with no forwarding
shell or alias. A1 moved root `cli.py` and
`arguments.py` to `apps/cli/__init__.py` and `apps/cli/arguments.py` and
deleted both root files with no forwarding shell or alias. Each later removal
PR shrinks the set in the same change: A2 (#408) moves the chart and the
benchmark launcher, and M1 (#452) moves the four MindIE files. The guard does
deleted both root files the same way. The last removal PR shrinks the set in
the same change: M1 (#452) moves the four MindIE files. The guard does
not assert the final two-file root (`__init__.py` and `__main__.py`) early,
and the build-generated `_version.py` is ignored wheel metadata, not a tracked
root file. [Retired root modules](#retired-root-modules) lists the eleven
paths that R1 removed and the two that A1 removed.
paths that R1 removed, the two that A1 removed and the two that A2 removed.

The current `src/inferrsi/harness/` tree is a mixed implementation. Its
presence is not the target Harness boundary and is not evidence that the four
Expand Down Expand Up @@ -107,7 +109,7 @@ Several flat modules remain shared seams until their slice starts.
| Current path | Tentative reading | This slice |
| --- | --- | --- |
| `apps/cli/` (`__init__.py`, `arguments.py`), root `__main__.py` | Apps CLI entry and its parser | A1 (#408) moved root `cli.py` and `arguments.py` here; the entry is `inferrsi.apps.cli:main`, and `__main__.py` only delegates to it |
| `benchmark_operations.py` | Single-task application launcher | Resolves a task entry and invokes the Harness runner; ordinary projects do not use it |
| `apps/benchmark_operations.py` | Single-task application launcher | A2 (#408) moved root `benchmark_operations.py` here; resolves a task entry and invokes the Harness runner, and ordinary projects do not use it |
| `benchmark/registry.py`, `benchmark/tasks/` | Task discovery, definitions and specialized rules | Task-specific quality and measurement code stays with each task |
| `harness/optimization_session.py`, `harness/project.py`, `harness/task_protocol.py` | Shared preparation, execution and retained-result assessment | CLI supplies task resources and knowledge access; ordinary project lifecycle does not import Benchmark |
| `harness/optimization.py`, `harness/candidates.py`, `harness/usage.py` | Shared project and packaged-task lifecycle | Injected execution/assessment; declared file/directory candidates, selection direction and exact/up-to rounds |
Expand All @@ -125,7 +127,7 @@ Several flat modules remain shared seams until their slice starts.
| `apps/inspection.py` | Prepared-environment inspection bound to this host | Binds the `infra/inspection.py` facts API to `infra.commands.probe_command` and `infra.targets.repository_identity`; replaces root `inspection.py`, which R1 deleted with the three old-run root forwarders |
| `model/validators/` | Benchmark-leaning checks | Not moved |
| `infra/` | Seams used by the current tree, including the `infra/inspection.py` facts API | Not a fourth core; reaches no Knowledge, Harness, Benchmark or Apps module |
| `summary_chart.py` | Current report helper | Not moved |
| `apps/views/summary_chart.py` | Stdlib SVG gain chart and report figure spec | A2 (#408) moved root `summary_chart.py` here; the renderer, its specification checks and its `main` are unchanged, and `tools/campaign_summary_chart.py` calls it |
| `contracts/` (`runspec.py`, `json_v1.py`, `claim.py`, `evidence.py`) | Technical-only contracts: RunSpec, canonical JSON, verdict vocabulary, evidence segment and path rules | Exact module set; the initializer re-exports only `RunSpec`; not a fourth core |
| `apps/evidence_layout.py` | Evidence directory creation over the `contracts/evidence.py` path rules | The `mkdir` half of former root `evidence_paths.py`; host I/O stays in Apps |
| `resources/repository_packs.py` | Neutral `REPO_TYPES` and `repository_pack` facade | Former root `_repo_pack_data.py`, bytes unchanged; import-free initializer; not a fourth core |
Expand All @@ -136,13 +138,12 @@ Several flat modules remain shared seams until their slice starts.
`tests/test_target_module_import_gate.py` fails on a new import of
`inferrsi.knowledge`, `inferrsi.benchmark`, or `inferrsi.apps` from anywhere
except application assembly, and on Knowledge or Benchmark importing Harness.
Application assembly includes `inferrsi.apps` and the one remaining flat
entry `benchmark_operations.py`, listed by exact file name as
`APPLICATION_ENTRIES` in `tests/_gate_policy_support.py`. A1 (#408) moved
`cli.py` and `arguments.py` into `apps/cli/`, which is Apps by its first path
segment, and removed their two entries, so a root file that returns under
either name is not admitted. The listing is temporary: A2 removes the last
entry. Other flat modules retain
Application assembly is the `inferrsi.apps` package, identified by its first path
segment. No root file is application assembly: A1 (#408) moved `cli.py` and
`arguments.py` into `apps/cli/`, A2 (#408) moved `benchmark_operations.py` into
`apps/`, and with that last move the exact-file `APPLICATION_ENTRIES` listing in
`tests/_gate_policy_support.py` is removed. A root file that returns under any of
those names is not admitted. Other flat modules retain
the same restrictions. The gate checks Benchmark task scripts as well as
module code.
`from inferrsi import knowledge` counts as an import of that core.
Expand Down Expand Up @@ -491,9 +492,21 @@ The `inferrsi` console script and `python -m inferrsi` call
unchanged. A test replaces a CLI name such as `_knowledge_reader` or `files` on
`inferrsi.apps.cli`.

The root now holds the eight files listed under
The root now holds the six files listed under
[Target modules](#target-modules). The final two-file root waits for the
A2 (#408) and M1 (#452) moves.
M1 (#452) move.

A2 (#408) deleted two more root modules the same way:

| Retired root module | Canonical module |
| --- | --- |
| `inferrsi.summary_chart` | `inferrsi.apps.views.summary_chart` |
| `inferrsi.benchmark_operations` | `inferrsi.apps.benchmark_operations` |

`tools/campaign_summary_chart.py` keeps its command line and calls
`inferrsi.apps.views.summary_chart:main`, so its arguments, exit codes, output and
the SVG and Markdown bytes it writes are unchanged. The dashboard, the HTML report
and the CLI `benchmarks run` command import the canonical modules.

## Old `run` exit codes

Expand Down
2 changes: 1 addition & 1 deletion docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ Effect classes used below:
| Optimize an ordinary project in the current Agent | CLI `campaign start/knowledge/round/status/final/recover` | Experiment-executing / local records | Harness freezes explicit caller-provided checks, manages multi-file candidates, executes and assesses retained results. This path does not depend on Benchmark or infer acceptance from the goal. See [shared workflow](optimization-workflow-unification.md) |
| Prepare and record a fresh native Agent | CLI `agents prepare/dispatch/bind/record/status` | Local records / explicit host handoff | The initiating Agent calls native host tools. Records prevent repeated dispatch; exported host observations do not establish task acceptance. See [native sessions](native-agent-sessions.md) |
| Discover packaged tasks | CLI `benchmarks list` / `benchmarks show`; Python `benchmark.registry.builtin_tasks` / `describe_task` | Read-only | Reads the task library's registry and opaque manifests from installed resources |
| Execute one packaged task | CLI `benchmarks run`; Python `benchmark_operations.run_task` | Experiment-executing | Application launches the task entry, supplies the execution callback and preserves arguments and exit status. The task owns its device, measurement and assessment protocol |
| Execute one packaged task | CLI `benchmarks run`; Python `apps.benchmark_operations.run_task` | Experiment-executing | Application launches the task entry, supplies the execution callback and preserves arguments and exit status. The task owns its device, measurement and assessment protocol |
| Search packaged records | CLI `knowledge search` / `knowledge show`; catalog API | Read-only | Installed package catalog with the existing tier/repo/backend/scenario filters and IDs; these package-only filters exclude skills and cases from every other authority |
| Prepare model-aware engine research | CLI `knowledge research`; Python `research_knowledge` | Read-only | Independent model/framework/task/modality/mechanism queries over the same package catalog; source-grounded known profiles, deduplicated matches and explicit target annotations; no engine/hardware qualification |
| Discover/read packaged or custom skills | CLI `skills list` / `show` / `resource` / `reference` | Read-only | Metadata-first listing; `show` loads one named instruction body; `resource`/`reference` resolve paths for your file reader without loading or executing them |
Expand Down
2 changes: 1 addition & 1 deletion src/inferrsi/apps/cli/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,7 @@ def main(argv: list[str] | None = None) -> int:
_print(result)
return 2 if result.get("state") == "blocked" else 0
if args.benchmark_action == "run":
from ...benchmark_operations import run_task
from ..benchmark_operations import run_task

return run_task(args.task, args.arguments)
from ...benchmark.registry import builtin_tasks, describe_task
Expand Down
2 changes: 1 addition & 1 deletion src/inferrsi/apps/legacy/reporting.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
from pathlib import Path
from urllib.parse import quote

from inferrsi.summary_chart import (
from inferrsi.apps.views.summary_chart import (
SpecError,
build_svg,
check_trial_chart_eligibility,
Expand Down
2 changes: 1 addition & 1 deletion src/inferrsi/apps/views/dashboard.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,11 @@
from pathlib import Path

from inferrsi.harness.records.evidence import inside, sanitized_text
from inferrsi.summary_chart import SpecError, build_svg

from .attempt_html import render_attempt
from .dashboard_data import _number, load_dashboard
from .dashboard_timeline import temporal_chart
from .summary_chart import SpecError, build_svg


def _sanitize_spec(value: object, depth: int = 0) -> object:
Expand Down
File renamed without changes.
Loading
Loading