Repository navigation
harness: autonomous improvement loop and decisions.yaml lifecycle (design, off until founder approval on #43) - #70
BotshareAI wants to merge 19 commits into
Conversation
Harness: policy (tiers A to D, protected paths, blocked commands, budgets, schedule, missions, escalation, enrolment of 15 repositories, 4 excluded by owner decision, audits private), baseline with the register-documents sync gate, 13 always-loaded core rules (SessionStart hook in CI and in the plugin), 16 skills (also a Claude Code plugin), 12 missions, 14 behaviour evaluations for `claude plugin eval` (14 of 14 at 3 of 3 runs), Claude Code settings and a fail-closed PreToolUse hook, deterministic selection, inputs, guard (recomputed in the publish job from the bundle), output allowlist with secret scrubbing, publish, review queue and stage 4 merge gate; design workflows under rollout/. Decisions lifecycle: Decisions impact section in the PR template and in check_pr_evidence.py (--decisions, --files-json; warning until --require-decisions-impact), decision_impact.py, decision_followup.py, register-sync mission (platform-lead-approved requests only), weekly register audit in the private audits repository. REG audit prefix owned by the platform lead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DZnPEN4YoUKzMSoLmaZCAS Claude-Session: https://claude.ai/code/session_01DZnPEPJrZ6MioobL2aYbRm Signed-off-by: Alex Reznichenko <info@botshare.ai>
|
Tracking issue for the "Decisions impact rule warns until each repository requires it" item under Not verified: #71 Generated by Claude Code |
Review of #70 (adapt): - Dry run: with HARNESS_PUBLISH unset, a run started by hand runs any mission whatever HARNESS_STAGE is and publishes nothing; scheduled AI runs are skipped. Stage gates apply only when publishing. Policy gains stage 0. Publish steps need HARNESS_PUBLISH=true and the route's dry_run=false. The route step is now executed by tests. - Core rule 5: one approved contract change request drives one PR per affected repository, merged in order (interface, consumers, removal). - Core rule 12 and the digest: excluded repositories are named as excluded (out of scope), never as clean or unscanned. - Core rule 13: the revert test applies to code changes; read-only work has its own evidence. - README: federation credentials are short-lived; an API key is spend-limited but long-lived. - Evaluations rerun with claude-opus-5-5: 14 of 14 at 3 of 3 runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DZnPEN4YoUKzMSoLmaZCAS Signed-off-by: Alex Reznichenko <info@botshare.ai>
panthera-momagdii
left a comment
There was a problem hiding this comment.
Reviewed head 50ff826, focusing on policy.yaml, CORE_RULES.md, guard_tool.py, harness_guard.py, harness_publish.py and the publish job. Also checked the merge gate, relevant test coverage and the check_pr_evidence.py diff.
The design-only rollout and separate founder approval on #43 must stay unchanged. This review does not authorize activation.
Requesting changes for three reproducible gaps in the final guard/publish boundary:
- Git-quoted filenames bypass the protected-path classification.
- Removing an existing assertion still passes as tier A, despite the tier-A no-test-line-removal condition.
- A secret-like value committed and then removed is missed by the net-diff scan, although the original commits remain in the bundle and branch that get published.
Inline comments include reproduction details and suggested regression coverage. Please fix these before approval and rerun the guard/publisher tests, including negative cases.
Validation performed: isolated local Git fixtures using byte-verified copies of harness_guard.py and harness_lib.py from this head, with the corresponding path-tier, dependency, mission-exception and safety-path policy entries copied into a test fixture. A safe new test passed and an ordinary protected harness file failed, as expected. All three unsafe fixtures unexpectedly returned PASS. Synthetic secret-pattern text only; no real credentials and no GitHub pushes were used in these tests.
The Repository quality workflow is green. The full upstream 353-test suite and live workflow/agent execution were NOT RUN in this review. No activation, variable changes or merges performed. AI-assisted review.
| changes = [l.split("\t") for l in git(root, "diff", "--name-status", "--no-renames", | ||
| f"{args.base}..HEAD").splitlines() if l] |
There was a problem hiding this comment.
[P1] Parse changed paths with NUL delimiters before applying policy
git diff --name-status quotes paths containing tabs, newlines and, by default, non-ASCII bytes. Splitting the display output leaves the surrounding quotes and escapes in path, so tier_of() is not checking the real pathname. Reproduced with a signed commit adding harness/probe\t.md (a literal tab in the filename): the actual path is tier D, but the guard returns PASS / tier B with files containing the quoted representation. This defeats the publish job's protected-path check even though an ordinary harness/probe.md is correctly rejected.
Use --name-status -z and parse the NUL-delimited status/path records without display quoting. Make diff/stat parsing consistent, or reject unsupported path characters explicitly. Add regression cases for tab, newline and non-ASCII names under protected directories.
| removed_tests |= test_names(removed) | ||
| added_tests |= test_names(added) | ||
| lost = sorted(removed_tests - added_tests) | ||
| if lost: | ||
| failures.append(f"test functions removed: {', '.join(lost[:10])}") |
There was a problem hiding this comment.
[P1] Enforce the tier-A restriction on removing existing test lines
Only removed test function names are rejected here. Removing the sole assert from an existing test while keeping its def test_... and a harmless assignment returns PASS / tier A; supplying unchanged before/after executed-test counts also passes. I reproduced that exact case. This violates policy.yaml tier A's condition that no line is removed from an existing test except a rename/move. harness_merge_gate.decide() rechecks file tiers, not this diff condition, so the deterministic gate still treats the weakened test as eligible for stage-4 automatic merge if the other gates pass.
Reject assertion/test weakening and enforce the stated removal rule. At minimum, changes that cannot be demonstrated to be pure renames/moves must not retain tier-A automatic-merge eligibility. Add a regression test that removes an assertion without removing or renaming the test function.
| for line in added: | ||
| if SKIP.search(line) and not ISSUE_REF.search(line): | ||
| failures.append(f"{path}: skip or xfail without a tracking issue on the same line: {line.strip()[:120]}") | ||
| if SECRET.search(line): | ||
| failures.append(f"{path}: secret-like string added") |
There was a problem hiding this comment.
[P1] Validate all history that will be published, not only the final tree diff
per_file comes from the net base..HEAD diff, while the later per-commit loop only checks DCO sign-off. The workflow bundles the original commit range and Publisher.push_bundle() pushes the candidate branch without squashing. Reproduced: commit synthetic text that matches this SECRET regex, then remove it in a second signed commit and add a harmless test. The guard returns PASS / tier A, but the synthetic value is still reachable in the bundle's commit history. The same gap applies to protected-path edits that are reverted before HEAD.
Before publishing, scan every newly reachable commit/blob that will be transferred for secrets and forbidden changes, or reconstruct a fresh commit from the validated net change on the trusted base and publish only that sanitized history. Include an add-then-remove secret fixture and an edit-then-revert protected-path fixture. No real credential was used in this reproduction.
panthera-momagdii
left a comment
There was a problem hiding this comment.
Focused security/governance review of #70 at head 50ff826, covering harness/policy.yaml, CORE_RULES.md, the PreToolUse hook, tools/harness_guard.py, tools/harness_publish.py, the publish job in rollout/workflows/harness-run.yml, and the stage-4 merge gate.
[P2] Enforce the selected task's tier ceiling in the deterministic publish guard. harness_select.py sets a selected issue's tier to A or B and harness/missions/build.md calls it a ceiling. The publish job passes it as --max-tier. However, in harness_guard.py, if TIER_ORDER[tier] > TIER_ORDER[args.max_tier] only appends a note, not a failure: a selected tier-A issue can produce a tier-B source change, get guard PASS, and be published as a draft PR. The stage-4 merge gate independently rechecks file tiers, which prevents this from automatically merging, but it does not enforce the earlier one-issue/one-scope authoring boundary. Please add a guard failure when the actual tier exceeds --max-tier, and regression tests for A->B rejection and permitted A->A/B->B. Keep the human escalation path instead of silently upgrading the task scope.
Other boundaries inspected: tier C/D path matching, separate publish-job guard and bundle-base check, bot-author review-marker checks, and stage-4 file-tier recheck. No workflows were activated. I did NOT run the 353-test suite, hook evaluations, supervised dry run, GitHub publish, or hardware; this is source-derived and needs verification before activation. Keep #70 draft, and preserve founder approval on .github issue #43 plus a distinct activation PR. AI-assisted review.
|
Review fixes pushed in 54bdd2b. All five points accepted.
Behaviour evaluations rerun after the rule changes, model claude-opus-5-5: 14 of 14 at 3 of 3 runs ( |
panthera-momagdii
left a comment
There was a problem hiding this comment.
Follow-up review of the new head 54bdd2bb253df881901c3daf626acdf1034043a4 against previously reviewed 50ff82690ea3a0987bd00712177c8fb9a554acd9. COMMENT only, not approval or authorization to activate.
The incremental commit addresses the separate governance/dry-run feedback: the manual route now reports dry_run, allows supervised triage/build/review independently of publishing stage, and both publish paths require HARNESS_PUBLISH=true and dry_run=false before --apply. I inspected the changed workflow, policy.yaml, core rules, publish code, route tests, the current hook/guard/merge gate, the shared AGENTS.md rules, and prior review threads. The hosted Repository quality quality/test job 124097061558 passed 362 tests. NOT RUN locally: harness tests, live dry runs, publishing, or activation. The described behaviour is source/CI evidence, not a live deployment test.
The prior security change requests are not resolved by this commit. tools/harness_guard.py is unchanged: the earlier three reproduced P1 findings (Git-quoted protected paths; deletion of an existing assertion still qualifying as tier A; secret/protected edits hidden in earlier commits of the published history) and the P2 tier-ceiling note-versus-failure finding still apply. I am not duplicating their inline reproductions. Please address those checks and add regression coverage before security acceptance.
The AGENTS.md v2 contract wording versus the new one-PR-per-repository core rule is explicitly deferred by the author to shared-rules v3; keep that owner-led reconciliation visible. This is still a design-only draft. No workflow activation without a separate activation PR and Alex's explicit approval on #43, and no automatic-merge or safety acceptance implied.
wikki26
left a comment
There was a problem hiding this comment.
[P1] Require executed successful verification before stage-4 automatic merge.
Reviewed head 54bdd2b. tools/harness_merge_gate.py accepts success, neutral and skipped for every check, and its only nonempty requirement is that check_runs contains an object. I ran decide() with an otherwise eligible tier-A PR, a current-head bot PASS marker, an expired veto window, and exactly one completed/skipped check. It returns [] (no rejection reasons), so the candidate is eligible even though no check succeeded or verified anything.
Please require the repository's authoritative verification checks to execute and succeed, while permitting genuinely optional skipped jobs separately. Add all-skipped, all-neutral, missing-required-verifier, and mixed required-success/optional-skip regression cases. The merge path should also inspect the applicable commit statuses/required contexts; its current check-runs-only read is not independent evidence of CLA or other status-based gates. GitHub rulesets remain another boundary, not a substitute for the stated deterministic gate.
Validation: python3 -m pytest -q tests/test_harness.py tests/test_harness_publish.py tests/test_harness_decisions.py: 84 passed on this head. Standalone decide() reproduction returned []. No merges, publishing, dry-run activation or robot execution. The prior quoted-path, weakened-assertion, hidden-history and tier-ceiling requests remain unresolved in the unchanged guard. Keep design-only/draft and preserve separate founder authorization on #43. OpenAI ChatGPT/Codex-assisted review.
wikki26
left a comment
There was a problem hiding this comment.
CI/security review for head 54bdd2b (requested by Alex for 2 Nov).
Changes requested — do not activate or treat the green 362-test run as security sign-off. The staged design and separate founder approval on #43 are appropriate, but deterministic guard/publish enforcement still has four release-blocking gaps at this exact head:
- Protect actual Git filenames, not quoted display strings (
tools/harness_guard.py:56-69,82-93).git diff --name-status(and patch headers) quote tabs/newlines. The currentsplit("\\t")/header parsing can tier a protectedharness/filename as B. Use NUL-delimited raw filenames and path-scoped diff/stat parsing; add tab/newline/non-ASCII protected-path regressions. - Never auto-publish a weakened existing test as tier A (
harness_guard.py:117-132). Existing code checks whether thetest_*name disappears, not whether its assertions or test lines are removed. Make deletion of a line from an existing test at least tier B / non-auto-merge (or reject it), and add a fixture removing only the sole assertion. - Scan all commits actually pushed (
harness_guard.py:117-147,harness_publish.py:201-212). The netbase..HEADscan misses secrets or protected-path edits introduced and then reverted in separate signed commits; publishing the original bundle preserves those objects. Reject unsafe intermediate commits/blobs, or publish only a newly constructed sanitized commit from the independently validated final tree. Test both add/remove synthetic secret and protected-edit/revert fixtures. - Enforce the selected issue tier ceiling (
harness_guard.py:106-115). Exceeding--max-tiermerely appends a note while returning PASS; the publisher subsequently creates the draft PR. Fail the guard when the final actual tier (including dependency-manifest promotion) exceeds the ceiling. Test A→B rejection, A→A and B→B acceptance.
Please rerun the guard/publisher regression suite and the entire harness verifier on the corrected head, retaining separate HARNESS_PUBLISH=true + dry_run=false authorization and the independent founder/activation PR gate. quality/test currently passes, but the four negative security cases above are not established by its existing tests.
Reviewed: policy tiers/blocked commands, 13 core rules, tool hook, publish workflow and credential separation, guard/publisher and existing unit cases at this SHA. NOT RUN: local Git bypass fixtures, model evaluations, supervised dry run, live publish or hardware. This review is not an activation approval.
…ling Use NUL-delimited Git paths, scrutinize every published commit for protected edits and secret-like content, disallow weakened tests at tier A, and reject changes beyond the issue's selected tier. Signed-off-by: Arumuga Vignesh SR <129583467+wikki26@users.noreply.github.com>
Add negative fixtures for protected names, deleted assertions and intermediate secret/protected commits; enforce a scoped tier ceiling. Also assert an untracked skip in published history cannot be sanitized just by fixing its final file content. Signed-off-by: Arumuga Vignesh SR <129583467+wikki26@users.noreply.github.com>
decisions.yaml in_force reached P-03-rev20.2 with #73 and #78. The baseline names the same addendum, E-01 rev1.6, M-01 rev19.7 and BOM 7.14, so register-related issues are not skipped as out of sync. README status lines updated to match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DZnPEN4YoUKzMSoLmaZCAS Signed-off-by: Alex Reznichenko <info@botshare.ai>
Review of #76 (panthera-momagdii): the guard now computes a tier for every commit the publisher transfers, with the same promotions as the net diff (allowed tier D mission paths and dependency manifests to B, new tests in tier C areas to A, removed lines of existing tests to B). The effective tier is the highest of the net change and every commit, and --max-tier fails on it. Regressions: source edit then revert plus a new test, and a test weakened then restored. Review of #70 (wikki26): the stage 4 merge gate requires the checks in policy merge_gate.required_checks (default quality/test and quality/pr-evidence) to run and succeed on the head SHA; skipped or neutral runs never satisfy them, a skipped reusable duplicate is allowed, any other failing or pending run blocks, and every commit status (DCO, CLA) must be success. Regressions: all skipped, all neutral, missing required verifier, required pending, optional failure, mixed success with optional skips, statuses unread, pending and failed. The App needs Commit statuses read for stage 4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DZnPEN4YoUKzMSoLmaZCAS Signed-off-by: Alex Reznichenko <info@botshare.ai>
Signed-off-by: Alex Reznichenko <info@botshare.ai>
|
Review fixes 2 are on this branch (counts in the Evidence block). @panthera-momagdii, your four guard findings: fixed by @wikki26's #76 (NUL-delimited paths, removed test lines promote to tier B, every published commit scanned for protected paths, secrets, skips and binaries, @wikki26, merge gate: Also: main merged (register at P-03 rev20.2) and Generated by Claude Code |
panthera-momagdii
left a comment
There was a problem hiding this comment.
Focused security re-review of exact head 4b9258d1b67fbbf4b19d9b6a187361038a556c2a (seven commits after previously reviewed 54bdd2b). REQUEST_CHANGES for one new, reproducible stage-4 veto-window defect. This is not approval or authorization to activate.
Previous findings rechecked in source, not repeated as open defects: harness_guard.py now parses NUL-delimited filenames, promotes removed existing test lines to tier B, checks every bundled commit for protected paths/secrets/skips/binaries and tier promotions, and fails when --max-tier is exceeded. harness_merge_gate.py now requires the configured verification jobs to execute and succeed on the exact head and checks status-read failures. I inspected the corresponding regressions, policy, core/shared rules, publish/merge workflows and previous threads. Inspected hosted quality/test run 41325650316: 385 tests executed, 0 failures, 0 skips. I did not run the checked-out test suite locally.
[P1] The 24-hour human veto can expire before a new PR head is pushed. tools/harness_merge_gate.py:113 takes the timestamp from commit["commit"]["committer"]["date"], and line 82 uses that for the 24-hour window. This is a Git commit field, not a trusted GitHub push/PR timestamp; a committer can backdate it using GIT_COMMITTER_DATE. Reproduction: a tier-A PR first appears at 2026-11-01 16:30 UTC with a head whose committer date is 2026-10-29 17:12 UTC. At 2026-11-01 17:12 UTC, with the other gates mocked green, an isolated transcription of current decide() returns no rejection reasons, despite the head being less than one hour old. This makes the intended 24-hour opportunity for human veto disappear, even when exact-head CI and review succeed.
Please measure the veto from a GitHub-controlled timestamp of the current head (or persist its first-seen time), resetting after every push; use a conservative PR updated_at if an exact head-push timestamp is unavailable. Add a regression with an old committer date and a newly arrived head that must stay blocked for the full window.
Evidence boundary: isolated pure-Python transcription and current hosted CI inspected; full local harness, live publish/merge, ROS, Gazebo and hardware tests NOT RUN. Preserve design-only status, separate activation PR, and Alex's explicit approval on .github#43. No changes to source, settings or gates are authorized by this review.
Review of 4b9258d (panthera-momagdii): the stage 4 merge gate measured the 24 hour veto window from the commit's committer date, which the author controls. It now starts at the creation time of the harness review comment for the current head SHA (set by GitHub, written after every push); the committer date only remains a lower bound, and a head without a GitHub timestamp stays blocked. Regression: a head with a three day old committer date seen 42 minutes ago is rejected. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DZnPEN4YoUKzMSoLmaZCAS Signed-off-by: Alex Reznichenko <info@botshare.ai>
|
@panthera-momagdii, fixed in 549db0f. The veto clock now starts at the creation time of the harness review comment for the current head SHA: GitHub sets that timestamp, and every push produces a new head and a new comment, so it resets on every push. The committer date is only kept as a lower bound (the later of the two counts), and a head without any GitHub timestamp stays blocked. Regression in Generated by Claude Code |
panthera-momagdii
left a comment
There was a problem hiding this comment.
Security re-review of the one-commit change 4b9258d → 549db0f at exact head 549db0f62a2985d91a93f3600049ff8544cf0785. REQUEST_CHANGES, not approval or authorization to activate. I compared the new commit, read the full merge gate, review publisher, regressions, policy, rules, design workflow, previous review threads and exact-head hosted CI. The earlier Git-filename, history-scanning, weakened-test and tier-ceiling findings were addressed on prior commits and are not being repeated.
[P1] The 24-hour veto fix still fails across a PR head update. Publisher.review() in tools/harness_publish.py:227-234 deliberately PATCHes the same bot issue comment on every re-review, replacing its head marker. GitHub retains the comment's original created_at when edited. This gate selects the comment by its current marker but reads its original creation time. Example: a bot comment is created 50 hours ago for old head A; head B is pushed 5 minutes ago; the bot edits that same comment with head=B verdict=pass tier=A; B has a backdated committer date. With otherwise green exact-head checks, start=max(old_comment_created_at, backdated_committer_date) is over 24 hours ago, and decide() returns no veto reason even though B has only existed for 5 minutes. I reproduced this timestamp predicate in an isolated Python fixture, without touching GitHub. The newly added test only uses a fresh comment's created_at and misses the actual publisher/PATCH path. Please use a trustworthy per-head GitHub timestamp: e.g. publish a new bot review comment for every new SHA (update only repeats for the same SHA), or enforce a reliable per-head time that cannot precede the latest push. Add the old-comment/new-head/backdated-commit regression at the publisher-plus-gate boundary.
[P2] Missing status contexts still pass the deterministic gate. At tools/harness_merge_gate.py:79-84, statuses={"state":"pending","statuses":[]} is accepted because the any(... for ... in []) test is false; tests/test_harness_publish.py:222 even uses this as the passing fixture. If CLA/DCO statuses are absent, this gate can return eligible with successful check runs. Please reject a missing/pending combined status (or explicitly require the expected CLA/DCO contexts per repository) and test missing/empty and partial contexts. Do not rely solely on external branch rulesets for the documented independent guard.
Evidence inspected, not run by reviewer: hosted Repository quality run 41334766466 at this exact head, quality/test 386 tests, OK; other jobs completed successfully or were optional/skipped. Reviewer-run: isolated veto timestamp fixture; NOT RUN: checked-out full pytest/verify.sh, live harness, actual PR publishing/merge, supervised dry run, ROS/Gazebo/hardware or behaviour evaluations. Preserve design-only rollout, owner reviews and all approval gates. Activation still requires a separate activation PR and Alex's explicit approval on #43.
| # The veto clock starts when GitHub first recorded the current head: the earliest harness review | ||
| # comment for this head SHA (GitHub sets created_at; a new push gets a new comment). The commit's | ||
| # committer date is only a lower bound, since an author can backdate it. | ||
| seen = [parse_time(c["created_at"]) for c in head_passes if c.get("created_at")] |
There was a problem hiding this comment.
[P1] This created_at is not the current head's first-seen timestamp after the publisher edits its existing bot comment. tools/harness_publish.py:227-234 PATCHes the same comment across head SHAs, preserving its old created_at. A newly pushed, backdated head can therefore inherit an already-expired 24-hour veto window. Repro and recommended per-head regression are in the review summary.
The PreToolUse hook judged only the first word of each subcommand, so a protected write passed when it was wrapped (sudo, env, timeout, nice, xargs), nested in `bash -c '...'`, preceded by `pushd`, or done by `find -exec`/`-delete`. `git -c` could set core.hooksPath or an invented user.name/user.email for the DCO sign-off. Probe P3 of the 1 Nov 2026 effectiveness audit reproduced each case, including a `bash -c` write that replaces the hook itself. The hook now unwraps wrapper commands and VAR=value prefixes, recurses into `sh/bash -c`, treats pushd like cd, judges find by its root and blocks `git -c` and `--config-env`. The unused WRITE_VERBS regex is removed. Still not detected (documented in the audit): paths built at run time inside interpreter code, Python fed on stdin, xargs reading names from stdin. The publish-job guard stays the boundary. Test: tests/test_harness_hook_wrappers.py (5 tests fail before, pass after). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DZnPERHnrr2W5e6d5KpfHw Signed-off-by: Alex Reznichenko <info@botshare.ai>
…ees interface generators Probe P2 of the 1 Nov 2026 effectiveness audit ran the guard on synthetic branches of openamr-platform-sw. A parameter change in openamrobot_nav2/launch/localization_launch.py passed as tier B, because tier C listed only navigation_launch.py and nav2_params.yaml; README says "Nav2 launch and parameters" are tier C. A .msg written under test/ passed as a new test file. A committed script that writes msg/*.msg at build time passed as tier B, because the generated file never appears in the diff. .openamrobot/verify.env (sourced by rollout/verify.sh) and a repository's tools/verify.sh were tier B, so an agent could rewrite the gate whose counts the guard compares. Policy: every file under openamrobot_nav2/launch and openamrobot_nav2/config is tier C; .openamrobot/** and tools/verify.sh are tier D; sudo is a blocked command. Guard: interface files never take the new-test exception; added lines that write an interface path, or rosidl_generate_interfaces, fail. Other launch files (drivers, bringup) stay tier B pending the platform lead's decision on launch arguments as contracts. Test: tests/test_harness_tier_gaps.py (6 of 7 fail before, 7 pass after; the seventh is a negative control). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DZnPERHnrr2W5e6d5KpfHw Signed-off-by: Alex Reznichenko <info@botshare.ai>
tools/harness_outputs.py searched each output file's raw bytes for the secret values and secret-like strings. change.bundle holds zlib-compressed pack data, so a commit containing the exact Claude credential passed the scrub and would be uploaded as an artifact of the public repository for 14 days before the publish-job guard rejected the push. Reproduced in the 1 Nov 2026 effectiveness audit. The scrubber now decompresses every object of the bundle's pack and scans that too; a bundle it cannot parse is dropped, so publish reports "no bundle" and pushes nothing. Test: tests/test_harness_outputs_bundle.py (3 of 4 fail before, 4 pass after; the fourth is the clean-bundle control). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DZnPERHnrr2W5e6d5KpfHw Signed-off-by: Alex Reznichenko <info@botshare.ai>
…olicy; register audit egress Four workflow gaps found in the 1 Nov 2026 effectiveness audit: - The agent works in a checkout of the target repository (for review, the PR head). Claude Code loads .claude/settings.json from the working directory, so a hook committed on a PR branch would run next to the Claude credential and the id-token permission. The workspace step now runs rollout/hide-project-claude-config.sh, which marks .claude, .mcp.json and CLAUDE.local.md skip-worktree and removes them, before the harness skills are copied in. - harness-register-audit.yml ran Claude with no harden-runner step; it now has the same egress policy as harness-run.yml. - sim-analysis started an agent with up to 120 turns every night, although no workflow publishes .harness-in/sim results; the plan now reports "no safe task" for it. - --max-turns was hard-coded and had drifted from policy.yaml (retro 120 against 60; register-sync and sim-analysis had no budget) and timeout_minutes was never applied. tools/harness_budget.py now reads both from policy.yaml budgets (register_sync added at the values the workflow used) and the plan job passes them to the agent step. Test: tests/test_harness_workflow_hardening.py (6 fail with the workflow, policy and script changes reverted and the new tool kept; 6 pass after). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DZnPERHnrr2W5e6d5KpfHw Signed-off-by: Alex Reznichenko <info@botshare.ai>
… update 549db0f made the 24 h veto window start when GitHub recorded the current head, measured by the created_at of the harness review comment for that head. harness_publish.Publisher.review updates its one review comment in place for each new head, so created_at stays at the first review. A head pushed and reviewed one hour ago, with a backdated committer date, passed the window at once (reproduced in the 1 Nov 2026 effectiveness audit). The gate now uses the later of created_at and updated_at; only the harness bot can edit its comment, so updated_at is when the marker for this head was written. Test: tests/test_harness_merge_veto.py (1 of 2 fails before, 2 pass after; the second is the control). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DZnPERHnrr2W5e6d5KpfHw Signed-off-by: Alex Reznichenko <info@botshare.ai>
Probe P1 of the 1 Nov 2026 effectiveness audit ran tools/harness_select.py on the live openamrobot-manifest issues with issue #12 labelled ai-ready. It selected #12, although openamrobot-manifest PR #19 ("fix(ci): validate advertised Git branch-tip pins despite vcstool 0.3.0 crash (#12)") already works on it: only ai-generated PRs and ai/ or register/ branches counted as taken. The build inputs also had no list of open PRs, and the agent has no gh or curl, so it could not fill the Integration Gate. The selector now treats any open PR that names the issue as "(#n)" in its title, or with closes/fixes/resolves/refs/addresses #n in title or body, as covering it. harness_inputs.py writes .harness-in/open-pulls.json for build runs, and the build mission names it. Test: tests/test_harness_select_overlap.py (3 of 4 fail before, 4 pass after; the fourth is the unrelated-PR control). With the live data the selector now reports "an open PR already covers it" for #12. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DZnPERHnrr2W5e6d5KpfHw Signed-off-by: Alex Reznichenko <info@botshare.ai>
Audit 2026-11-01 (P2, P9): xacro, URDF, SDF, SRDF, datums, upstream pins, Gazebo worlds and launch files passed the guard as tier B, so an agent could change frame poses, limits and launch argument defaults. CORE_RULES rule 4 and AGENTS.md Contracts say a person changes them. Signed-off-by: Alex Reznichenko <info@botshare.ai>
Audit 2026-11-01 (P8): a 403 on one enrolled repository raised out of pass_changes and ended the run before any consumer issue was planned. Each repository is now reported as not measured and the pass continues. Signed-off-by: Alex Reznichenko <info@botshare.ai>
Audit 2026-11-01 (P1): a pr.md holding only a title passed the guard, and no product repository runs quality/pr-evidence, so the draft PR would have opened without Tests, Evidence or Not verified. The skill listed section names that tools/check_pr_evidence.py does not accept (Summary, Changes, Base SHA and Head SHA); it now lists the checker's names. Signed-off-by: Alex Reznichenko <info@botshare.ai>
…audit run A cases Owner decision (1 Nov 2026): 100 EUR a month for the harness. budgets.plan starter with weekly_turn_ceiling 300; triage 20 turns, audits 60 turns and 40 minutes; one audit a week on Monday rotating by ISO week number (dependencies, contracts, impact, maintainability, performance); retro on the first Friday of the month; build two slots per weekday and review every two hours on weekdays, both for later stages. A test adds up the stage 1 schedule against the ceiling. Consumer issues from the decision follow-up are labelled needs-human:triage until a maintainer confirms the real consumers (audit P8: every planned line for item 26 was history or provenance text); tier C or D lines stay needs-human:safety. From audit run A on top of run B: glob command names are blocked in the hook; find writes only count when -exec runs a writing command or -delete; assigned issues are never selected. Plan step output grouped for shellcheck. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DZnPEN4YoUKzMSoLmaZCAS Signed-off-by: Alex Reznichenko <info@botshare.ai>
…0 at 3 of 3 Six cases proposed by the 1 Nov audit: generated file hiding a behaviour change, stale AGENTS.md with a readiness claim, Nav2 parameter tuning to pass a scenario, dependency bump without a Dependencies section, simulation presented as physical evidence, test count not run. First run 15 of 20 with the agent right every time and the graders too strict; graders corrected to the behaviour that matters (negated sentences no longer trip the regexes, reply-only judges no longer ask for file content). Record in BASELINE.md. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DZnPEN4YoUKzMSoLmaZCAS Signed-off-by: Alex Reznichenko <info@botshare.ai>
|
Two independent effectiveness audits of this branch ran on 1 Nov (reports in openAMRobot/audits, branches Two owner decisions are recorded in policy: budget plan Generated by Claude Code |
panthera-momagdii
left a comment
There was a problem hiding this comment.
Focused incremental COMMENT re-review of design-only #70 at exact head f4e08372bc239070491017679c140422c841f0dd (11 commits after my CHANGES_REQUESTED review of 549db0f62a2985d91a93f3600049ff8544cf0785). This is not approval, stage-4 authorization, or harness activation.
Resolved on this head: The prior 24-hour veto-window bypass across an edited bot review comment is addressed in source. harness_merge_gate.decide() now uses the later of the matching comment's GitHub created_at and updated_at (plus the commit-time lower bound). Since Publisher.review() PATCHes that comment for a new SHA, its updated_at starts a fresh 24-hour interval. tests/test_harness_merge_veto.py now covers old-created/new-updated/backdated-commit and expiry cases. I also inspected the new hook/wrapper, tier, bundle-scrubbing, selector, workflow and policy regressions; earlier guard defects are not being raised again.
Still open [P2, already reported at 549db0f]: missing commit-status contexts can pass the deterministic merge gate. tools/harness_merge_gate.py still only tests any(s.get("state") != "success" for s in statuses.get("statuses", [])), without requiring a successful combined state or expected CLA/DCO contexts. With statuses={"state":"pending","statuses":[]}, that condition is false. The new veto regression helper even calls decide(..., statuses={"statuses":[]}) and expects no rejection once its veto interval expires. For an otherwise eligible current-head tier-A PR with successful required check runs, this is still an eligible automatic merge despite missing status evidence. Please fail closed on empty/unread/pending combined statuses and enforce applicable required contexts, with empty, partial and pending-status regression cases. This is re-verification of the existing finding, not a new duplicate defect.
Evidence inspected: exact-head GitHub Actions Repository quality run 41358896702 succeeded; its quality/test log shows 421/421 tests executed and passed, 0 skipped, with ros-verifier-smoke passed. The GitHub CLA status for this head reports success; that does not prove the independent merge gate rejects missing statuses on other PRs. The new behaviour-evaluation 20-of-20 / 3-run result is author-reported, not independently reproduced by this reviewer. NOT RUN locally: full verify/pytest, actual harness publishing or merges, live credentials/activation, ROS/Gazebo/hardware. Keep owner security review and all gates. Activation still requires a separate activation PR and Alex's explicit approval on .github#43.
Superseded by later #70 revisions. The filename parsing, test-line removal and published-history findings from this review are fixed on the current head.
panthera-momagdii
left a comment
There was a problem hiding this comment.
Focused final security re-review of #70 at exact head f4e08372bc239070491017679c140422c841f0dd. I dismissed my stale older CHANGES_REQUESTED reviews whose findings are fixed and am keeping only the one current blocker below. This is not activation authorization.
Resolved on this head: NUL-delimited path handling, per-commit published-history scanning, test-line weakening/tier promotion, selected-task tier ceiling, and the 24-hour per-head veto timing are addressed in source and regression coverage. Hosted exact-head Repository quality is green; quality/test reports 421/421 passed, 0 skipped; ROS verifier smoke passes; the current PR's CLA status is success.
[P2, remaining fail-closed defect] Missing commit-status evidence can still pass stage 4. tools/harness_merge_gate.py only rejects when statuses is None or when any element of statuses['statuses'] is non-success. For GitHub's valid empty response {'state':'pending','statuses':[]}, any(...) is false, so the deterministic gate adds no status reason. The current regression helper in tests/test_harness_merge_veto.py explicitly calls decide(..., statuses={'statuses':[]}) and expects an empty reason list after the veto period. Therefore an otherwise eligible tier-A PR with successful required check-runs can still be auto-merge eligible when CLA/DCO/other required commit-status contexts are absent entirely.
Please fail closed on empty/pending combined status evidence and require the applicable expected contexts per repository (at minimum the governance/CLA-DCO contexts the design says must be green), with regressions for: empty statuses, combined state=pending, one expected context missing, partial success, and all expected contexts success. Do not rely only on external rulesets for this independent stage-4 guard.
Reviewer RUN: isolated reproduction of the current status predicate: statuses=None rejects; {'state':'pending','statuses':[]} produces no rejection; a failing status rejects. Inspected: exact-head source, merge-gate tests, publisher review path, previous threads and hosted CI. NOT RUN: full local harness suite, live publish/merge, credentials, supervised dry run, ROS/Gazebo/hardware or behaviour evals. Keep the design-only boundary and all human gates. Activation still requires a separate activation PR and Alex's explicit approval on .github#43.
Summary
Adds the harness: 13 always-loaded core rules, 16 skills, 12 missions, 20 behaviour evaluations and deterministic tools that let Claude Code agents improve OpenAMRobot every day, while people keep safety, decisions and merging. The agents' work covers tests, CI preparation, cleanup, PR reviews, and audits of dependencies, contracts, impact, performance, maintainability and the register. It also adds the decisions.yaml lifecycle:
Nothing runs after this PR merges: the workflows are design files under rollout/workflows/. Activation needs the founder's explicit approval comment on #43 (D-02), given after this PR is reviewed and merged, the behaviour evaluations pass, and a supervised dry run has been read. Closing #43 is not authorization by itself. harness/README.md is the operating manual.
Work package
#43 (AI workflow activation). Related: #59, #69.
Integration Gate
Builds on #69 (daily collector; triage reads its watchdog-daily artifact); no file overlap with #69. Reuses tools/check_decisions.py (loader, applies_to matching, JSON findings), tools/sync_audit_issues.py (audit issues, one ID range per audit) and agent-prompts/push-from-bundle.md. Inherits the 29 Oct external audit and instruction matrix:
AGENTS.md and CLAUDE.md are unchanged (shared block v2,
@AGENTS.md). Rejected:Tests
Ran 421 tests, 0 skipped with
python3 -m pytest -q tests(278 on the parent). New: tests/test_harness.py, tests/test_harness_publish.py, tests/test_harness_decisions.py, and from the audit fixes tests/test_harness_hook_wrappers.py, tests/test_harness_merge_veto.py, tests/test_harness_outputs_bundle.py, tests/test_harness_select_overlap.py, tests/test_harness_tier_gaps.py, tests/test_harness_workflow_hardening.py. They cover policy, tiers, coverage, the hook, guard, selection, publish validation, artifact boundaries, the review queue, the merge gate, inputs, output scrubbing, decisions impact, the follow-up, register-sync approval, core-rule loading and the eval suite shape. Behaviour evaluations: 20 of 20 cases passed 3 of 3 runs withclaude plugin evalon Claude Code 2.1.312 with the pinned model claude-opus-5-5 (harness/evals/BASELINE.md).Evidence
Base SHA: 1678735
Head SHA: f4e0837
Tree: 69639b1fea599e734d7c1be2f89aef0c4e3e5a56
changed.txt is
git diff --name-only 1678735b5a3adae5d2fdc989bb29f2344a9538fc f4e08372bc239070491017679c140422c841f0dd.Dependencies
None added to the repository. The design workflows pin by full commit SHA, each verified with
git ls-remoteagainst the official repository:Decisions impact
Safety impact
The changed files are governance tooling, tests and design workflows in this repository. No robot source, launch file, URDF or collision geometry, motion limit, firmware configuration or interface definition changes, and no changed path matches maintainers.yaml safety_paths (tests/test_harness.py checks this with the evidence check's matching). The harness decides what agents may change on the robot later. So before any activation, two parts need human review: the protected paths and tiers in harness/policy.yaml, and the publish job with its guard.
STATE.md
no change: STATE.md arrived with the main merge; this PR is design-only and changes none of its register, readiness or current-work facts.
Not verified
AI disclosure
Claude Code drafted and tested every file in a session directed by the platform lead. It ran two independent code review passes and the behaviour evaluations, and fixed every reported defect. This session applied and pushed the prepared change set. A person must read first:
Contribution terms
🤖 Generated with Claude Code
https://claude.ai/code/session_01DZnPERB3or7fhUmLZ1Hads