Skip to content

harness: explain gains with profile evidence and paired statistics - #457

Draft
BeatSeat wants to merge 9 commits into
InferMatrixLab:mainfrom
BeatSeat:feat/paired-measurement-statistics
Draft

BeatSeat wants to merge 9 commits into
InferMatrixLab:mainfrom
BeatSeat:feat/paired-measurement-statistics

Conversation

@BeatSeat

@BeatSeat BeatSeat commented Nov 15, 2026 •

Copy link
Copy Markdown

Governing Issue and delivery slice

Refs #454

  • Accepted scope / delegation: rfc: 可解释的改进——用 profile 证据和配对统计判定 PASS #454 已由 steward 代 Dong1017 接受方向(附条件):可选能力、两个开关默认关闭、按一个 PR 评审;硬件 A/A 不是合入前提(决定)。本 head 处理了全部条件和四位评审的意见,见"评审后的修订"。
  • What this PR delivers: 普通项目优化(campaign start/round/final)新增三项能力,默认都不改变现有判定:
    1. 配对统计(默认只报告):按 steps() 已有的 AB/BA 交替,把同一次重复里的基线和候选样本配对,报告逐对加速比中位数的精确次序统计量区间(符号检验反演,至少 6 对),以及区间与门槛的关系。
    2. 用 profile 解释改进:测量脚本可以额外输出 breakdown(分区域耗时)和 counters(工作量计数);每轮可以用 --mechanism-file 声明机制和预期方向;Harness 按声明条数做 Bonferroni 校正后机械核对,给出每条核对的 outcome 和 explanation.status。
    3. 两个冻结开关:--require-explanation 只采纳 explained 的候选;--decision paired 用配对区间判定,区间跨门槛判 INCONCLUSIVE(within_noise),不被采纳也不阻塞搜索。两者同时打开时,先由 --decision 得出性能结论,--require-explanation 只降级 PASS。
  • What it does not deliver: 默认判定变更;基准题包支持(benchmarks session 拒绝 --mechanism-file);Dashboard 展示;与具体 profiler 绑定的采集脚本;GPU/NPU 实测。

Knowledge authorship declaration

本 PR 不触及 knowledge-model policy 的受保护路径,预期为 Not applicable。

{
  "schema_version": 1,
  "head_sha": "6a3c123691b0ba6937084837c62a888b9735f59b",
  "human_only": false,
  "material_ai_authors": [
    {
      "tool": "Claude Code",
      "model_id": "claude-opus-5-5",
      "effort": "unknown",
      "role": "author"
    }
  ]
}

Changed artifacts and evidence

文件 改动 作用
src/inferrsi/harness/measurement.py 新增 359 行 精确次序统计量区间(interval_rank、median_interval、min_pairs,二项尾用有理数精确比较)、配对统计、profile_evidence(Bonferroni)(原始与归一化方向相反时记为 raw_normalized_conflict)、validate_mechanism、explain;无法计算的值一律返回 unavailable,记录保持严格 JSON
src/inferrsi/harness/project.py +68/−5 assess() 接入两个开关和机制核对;profile 报告出错时记为 unavailable,不影响判定;新增设置项的校验
src/inferrsi/harness/optimization.py +32/−2 在执行前把声明和摘要写入 pending;settle 时核对摘要,并把内存中的声明传给判定器;final 使用被选中轮次绑定的声明
src/inferrsi/harness/optimization_session.py +15/−6 冻结两个开关;读取并校验 --mechanism-file;_runtime_identity 纳入 measurement.py
src/inferrsi/apps/cli/arguments.py +9 --decision、--require-explanation、--mechanism-file
docs/optimization-workflow-unification.md +36 中文说明:字段、取值、精确区间与 Bonferroni、开关与优先级、局限
tests/test_harness_measurement.py 新增 91 个用例(含参数化) 精确秩(n = 1…40 逐一核对最大性)、最少对数与 Bonferroni 所需对数、门槛处 A/A 误报不超过精确上界、上千对不溢出、方向判定、零基线、参照区域不互证、unchanged 只记"未推翻"、原始与归一化方向相反不被读成任一方向、机制校验与分类
tests/test_project_explanation.py 新增 21 个用例 端到端:确认的机制在 final 复核;不可采纳情形都让搜索继续;开关关闭时只报告;within_noise 为 INCONCLUSIVE;两开关组合 6 例;点估计未过而区间过门槛判 PASS;profile 报告出错不改判;篡改 state.json 或副本;中断后伪造声明的恢复
tests/test_project_optimization.py、tests/test_optimization_session.py +4、+1 个用例 配对统计不改判、不可达门槛、旧记录可读;题包会话拒绝机制声明
tools/measurement_simulation.py 新增 126 行 可复算的合成消融;--gate-pct 0 给出门槛处 A/A 比例与精确上界

判定逻辑

explanation.status 含义 --require-explanation 下
explained 每条预期变化都 confirmed,且没有任何一条 contradicted 可采纳
unsupported 有预期变化 not_shown(未检出;区间仍可能包含预期的变化) INCONCLUSIVE,继续搜索
contradicted 方向相反,或声明 unchanged 的项检出变化 FAIL,继续搜索
unverifiable 缺少数据、比值无定义、原始与归一化方向相反、对数不足以支撑校正后的置信度(k = 1 需 6 对,2–3 需 7 对,4–6 需 8 对),或证据按更少条数校正 INCONCLUSIVE,继续搜索
not_declared 本轮没有声明机制 INCONCLUSIVE,继续搜索
--decision paired 区间与门槛 verdict / reason
above_gate PASS / paired_interval_above_gate(即使点估计未过门槛;此时 agrees_with_point_gate: false)
straddles_gate INCONCLUSIVE / within_noise(没能证明过门槛,不是门槛未达到的证据)
below_gate、gate_unreachable FAIL / performance_gate_not_met
统计不可用 INCONCLUSIVE / paired_statistics_unavailable,不退回未配对规则

在 --require-explanation 下只有 contradicted(数据与预期相反)记 FAIL,证据不足的情形都记 INCONCLUSIVE,与配对性能判定的原则一致。若某区域原始比较和"除以参照区域"后的比较给出明确相反的方向(例如原始增加 20%、相对参照却下降),记为 raw_normalized_conflict,该条核对为 unverifiable:机制声明无法表达"只预期相对变化",两个方向都不采信。

声明为 unchanged 的项只会被记为 not_contradicted(未检出变化)或 contradicted,不会被"确认不变":没有等价性容差时,区间包含 1 不是"没变"的证据。声明为 unchanged 的区域同时是本轮的假设:其他区域会另外给出"除以同一步骤参照区域"后的比较,share_based_speedup 用参照占比推算整体加速。只有让一个步骤内所有区域按同一比例缩放的变化,才会被这样抵消。参照区域本身只做原始比较,不会互相归一化。

合成消融(python tools/measurement_simulation.py,CPU 仿真,非设备实测)

每格 200 次 campaign,数据 seed 20261115;不配对区间这一对照规则用 bootstrap 10000 次。200 次 campaign 的 Monte Carlo 标准误约 0.01–0.03,门槛处的误报以下方精确上界和大样本复核为准。

  • 门槛 1%,A/A: 现行规则判 PASS 的比例为 0.200–0.380;只配对、不算区间为 0.185–0.390;配对精确区间 ≤ 0.010(≥ 6 对)。压低误报的是区间,不是配对。
  • 门槛 0%,A/A(真实中位数恰好在门槛上): 现行规则约 0.45–0.57;配对精确区间 ≤ 0.030,与精确单侧上界(6/7/10/13/20 对分别为 1.56%/0.78%/1.07%/1.12%/2.07%)在 Monte Carlo 误差内一致。Codex 每格 50,000 次复核:6/7/10/13 对的误报为 1.55–1.65%、0.72–0.82%、1.06–1.10%、1.12–1.17%,与精确上界的差距都在 1.56 个标准误以内。
  • 检出力(真实 1.05×,lognormal,20 次重复,门槛 1%): 配对区间 0.880,不配对区间 0.575。配对带来的是检出力。 精确区间比原来的 bootstrap(0.930)略保守,这是换取任何对数下误差都有保证的代价。
  • 5 对及以下不给区间:最极端的一对也有 1/32 ≈ 3.1% 的单侧误差,超过 2.5%。

Checks and dependencies

  • Checks actually run (result and tested scope):
    • ruff check src tests tools:通过。
    • 新 head(基于 main 6c7e2379)上运行 34 个 harness / campaign / project / dashboard / optimization 相关测试文件(含 tests/test_harness_measurement.py、tests/test_project_explanation.py、tests/test_project_optimization.py、tests/test_optimization_session.py、tests/test_module_boundaries.py 和 Dashboard 测试):全部通过,无失败。Git 相关测试使用临时 Git 2.49.1(本机 2.43 不支持 --no-lazy-fetch)。
    • 未重跑本地全量 pytest,以 hosted CI 为准。
  • 独立评审: Codex CLI(gpt-6.1-sol,xhigh)本地只读评审,每一轮都用对抗输入去复现问题。
    • 前六轮发现并已修复:门槛 100% 或比值溢出时写出 Infinity;零基线计数在 1 对时就给出确定方向;breakdown 全为零时抛异常;多个参照区域互相归一化;候选在执行期间改写 mechanism.json 或 state.json 中的声明;中断后经 recover 接受伪造的声明(现记录 mechanism_unverified_after_recovery,按未声明处理)。
    • 针对本轮修订的第七轮:精确秩在 n ≤ 300 的 10,800 组有理数检查中无偏差;离散值和并列时区间保守;88 种判定组合和 1000 组默认行为对照通过;未发现依赖已删除字段或旧语义的代码。发现一处 P2:上千对时 2 ** n 转浮点溢出,且 profile 报告异常会改变默认判定。已修复(有理数精确比较;报告出错记为 unavailable)并补回归测试。第八轮复核结论:Ready(1024 对复现通过;注入报告异常后默认判定不变;31,255 组精确秩检查无偏差)。
    • 针对 david6666666 意见的第九轮:原始与归一化方向相反在单参照、多参照下都被拦截,另构造的单参照反例也判为 unverifiable;8,000 组方向组合检查无反向误判;原始样本不足或零基线不会经归一化变成明确方向;assess 与独立 final 都是 unsupported → INCONCLUSIVE、contradicted → FAIL,搜索继续;rebase 后无旧 CLI 路径残留。结论 Ready with fixes,唯一一项是同步本 PR 说明(即本次更新)。
    • plan.json 已有 plan_digest 保护。
  • Not run: hosted CI;GPU/NPU 实测;真实 profiler 采集。
  • Dependencies / integration assumptions:
    • measurement.py 只用标准库。
    • 配对依赖 steps() 两侧等长且交替执行;区间的精确性依赖各次重复独立、比值来自连续分布,离散计数出现并列时偏保守。
    • 候选以调用者权限运行,哈希核对是漂移检测,不是沙箱。
    • 修改 harness/*.py 会改变 runtime identity,旧 runtime 启动的会话需要用原来冻结的 runtime 恢复(既有规则)。

评审后的修订

来源 意见 处理
steward 条件 1;amy-why-3459;cursor[bot] rebase 到当前 main 已 rebase 到 6c7e2379(含 #461 的 CLI 迁移),无冲突;新 head 上重跑测试
steward 条件 2;amy-why-3459;cursor[bot] 两开关优先级;点估计未过而区间过门槛 写入文档和 #454;test_both_switches_apply_the_decision_before_the_explanation(6 例)、test_paired_decision_passes_when_only_the_point_estimate_misses_the_gate
steward 条件 3 默认关闭、只报告 保持;新增 profile 报告出错不改判的回归测试
linyueqian bootstrap 下界是次序统计量,5/7/13 对单侧误差 3.1%/6.25%/4.6% 改为精确次序统计量区间,最少 6 对;记录 order_statistic_rank、achieved_confidence
linyueqian unchanged 不是证据,且没有多重比较校正 Bonferroni(family_size、confidence、pairs_needed);unchanged 只记 not_contradicted / contradicted
linyueqian 跨门槛记 FAIL 与 docs/contributing.md 的 FAIL 定义冲突 改为 INCONCLUSIVE(within_noise);低于门槛仍为 FAIL
linyueqian 区间决定判定时仍写 report_only: true --decision paired 时为 false
david6666666(行内) 原始与归一化方向相反时,归一化方向覆盖原始方向,原始增加 20% 被判为"已解释的下降" 记为 raw_normalized_conflict,该条核对 unverifiable;test_raw_and_normalized_sign_reversal_is_not_read_as_either_direction 复现了他描述的 10 对情形(两个参照各自跨过 1,但总和每对都上升),修复前判 explained,修复后为 unverifiable
david6666666(#454) unsupported 是证据不足,应与 contradicted 区分 --require-explanation 下 unsupported 改为 INCONCLUSIVE,只有 contradicted 记 FAIL;候选都不被采纳

Remaining work and Issue closure

Copy link
Copy Markdown
Collaborator

我对照当前 main cd8fcb452bd2c7969a7a6e2544aa1cb5f8ad5332 和本 PR 的 head 1b2b4593cb72ef7b956f82f000313a9910cffbd7 做了完整 review(分支点是 a92c77ccaed68426c83154cff7619745f34d0058,其上 5 个提交:9e01b92bf08e505c259b948633d976019936dcac … 1b2b4593cb72ef7b956f82f000313a9910cffbd7)。#454 仍是 Proposed,这条评论只作设计/实现对照,不是 Approve。

先说已经对上我在 #454 里提的几条:explained / unsupported / contradicted / unverifiable 的机械定义在 explain() 和文档里写清楚了(全部确认才算 explained;任一条方向被打脸即为 contradicted;少于 5 对或零基线计数为 unverifiable);默认关闭时,test_assessment_reports_paired_statistics_without_changing_the_verdict 和 test_explanation_is_report_only_without_the_switch 覆盖了原判定不变;机制在执行前绑定,并覆盖了改写 state.json / mechanism.json,以及 recover 后不信任声明;tools/measurement_simulation.py 已附上,设定(data seed 20261115、门槛 1%、200 次 campaign、bootstrap 10000、seed 0)可以复核。steps() 已按重复交替 AB/BA,completed 的 assess() 两侧样本等长;某一侧测量失败会按既有 incomplete_execution 整轮失败,--decision paired 在统计 unavailable 时是 INCONCLUSIVE,不会退回未配对中位数。

还需要处理的点:

  1. Governing issue 尚未接受。 rfc: 可解释的改进——用 profile 证据和配对统计判定 PASS #454 仍是 Proposed,goal-acceptance owner 为 TBD,三个 owner 决策还没有结论。在 issue 被接受之前,这个草稿适合用来对照设计,但我不会 Approve。另外,rfc: 可解释的改进——用 profile 证据和配对统计判定 PASS #454 正文里还没有补上本 PR 的链接,请补上 https://github.com/InferMatrixLab/InferRSI/pull/457。

  2. 请 rebase 到当前 main。 分支点是 a92c77ccaed68426c83154cff7619745f34d0058,之后 main 已合入 Package shared technical contracts and retire root modules #453(cd8fcb452bd2c7969a7a6e2544aa1cb5f8ad5332)。Package shared technical contracts and retire root modules #453 改了 src/inferrsi/arguments.py 的 RunSpec 导入路径(本 PR 也改了这个文件,hunk 看起来不重叠,但仍需要先 rebase)。正文里本地全量 pytest 的对比基线也是 a92c77cc;rebase 后请按新 head 再跑一遍定向测试,并视情况更新声明里的 head_sha。

  3. 两个开关同时打开时的优先级,以及点估计与区间不一致时的判定,请写进文档并补测试。 这是我在 rfc: 可解释的改进——用 profile 证据和配对统计判定 PASS #454 评论里第 3 点仍未钉死的部分。assess()(src/inferrsi/harness/project.py 第 188–214 行)先按 --decision 得到 verdict,再仅在 verdict == "PASS" 时应用 --require-explanation。因此两者同时打开时,若配对结果已经是 FAIL / within_noise,即使 explanation 是 unverifiable,最终仍是 FAIL,不会变成 INCONCLUSIVE。请在 rfc: 可解释的改进——用 profile 证据和配对统计判定 PASS #454 和 docs/optimization-workflow-unification.md 里写明这条优先级,并加一个同时打开两个开关的回归测试。另外,--decision paired 只看区间(above_gate 即为 PASS,第 192–193 行),所以“点估计未过门槛、但配对区间整体越过门槛”时会判 PASS;这一点目前只是从代码读出来的,请在文档里写一句,并最好补一个 agrees_with_point_gate is False 且区间为 above_gate 的用例。

我没有重新跑 tools/measurement_simulation.py 的 200×10000 仿真,也没有跑 hosted CI。硬件 A/A + 已知改进对照仍是 #454 留给 owner 的决策,不作为这条评论的合入前提。默认关闭、机制绑定和合成测试设计看起来是对口的。

我不会在这个 head 上 Approve,直到 #454 被接受,并且上面的 rebase / 文档与测试缺口处理完毕。

@Dong1017 Dong1017 added area:harness Harness module: profile, A/B loop, quality gates, evidence reports kind:task Bounded delivery task with a delegated slice labels Nov 15, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS
Reviewed base cd8fcb452bd2c7969a7a6e2544aa1cb5f8ad5332 and head 1b2b4593cb72ef7b956f82f000313a9910cffbd7 (draft).

Acting reviewer: cursor[bot] via standing InferRSI steward delegation. Binding formal review to current head.

Blockers before Approve / ready-for-review:

  1. Governing issue not accepted. #454 is still Proposed with goal-acceptance owner TBD. Steward decision will be posted on #454; do not treat this draft as authorized delivery until that record exists.
  2. Rebase onto current main. Head diverges: ahead 5 / behind 1 vs cd8fcb45 (#453), which also touched src/inferrsi/arguments.py. Rebase and re-run the focused harness tests on the new head; refresh declaration head_sha if needed.
  3. Dual-switch priority and point/interval disagreement are under-specified in docs/tests. In assess() (project.py), --decision runs first; --require-explanation only rewrites a PASS. So with both switches on, within_noise FAIL stays FAIL even if explanation is unverifiable. Also, paired mode keys only on gate_relation (above_gate → PASS), so a point estimate below the gate can still PASS when the interval clears the gate. Document both behaviors in #454 and docs/optimization-workflow-unification.md, and add regression tests (both switches on; agrees_with_point_gate is False with above_gate).

Checked / helpful but not enough to Approve: mechanical explanation.status definitions look consistent with the RFC; defaults-off report-only path is covered; mechanism binding / tamper / recover handling is present; tools/measurement_simulation.py exists with stated seeds. I did not re-run the 200×10000 simulation or GPU/NPU hardware checks. Hosted CI on this draft is limited (declarations green); full ready-lane evidence is still missing.

Hardware A/A + known-gain confirmation is not a merge prerequisite while both switches default off (steward decision on #454). Re-request review after acceptance + rebase + the doc/test gaps above.

Copy link
Copy Markdown
Collaborator

回看了 Dong1017 在 #454 的 steward 决定:三个 owner 决策已经给出(可选能力、默认关闭、按一个 PR 评审),goal-acceptance owner 记为 Dong1017,硬件 A/A 也不再作为合入前提。这一条外部阻塞已经解除。

当前 head 仍是 1b2b4593cb72ef7b956f82f000313a9910cffbd7,这次没有新的提交,所以我上次列出的其余阻塞还在:

  1. 仍需 rebase 到当前 main。 分支点还是 a92c77ccaed68426c83154cff7619745f34d0058。在 Package shared technical contracts and retire root modules #453(cd8fcb452bd2c7969a7a6e2544aa1cb5f8ad5332)之后,main 又合入了 refactor(C2): relocate resource facade and split evidence layout #458(5db1bc94042f386f0b4b35c3e6fe91598fef118f)。请变基到这个新的 main,按新 head 再跑定向 harness 测试,并视情况更新声明里的 head_sha。
  2. 两个开关的优先级、以及点估计与区间不一致时的判定,仍未写入文档并补测试。 这次 head 没有改动,缺口与我在 1b2b4593cb72ef7b956f82f000313a9910cffbd7 上的上次评论相同。

因此我不会在这个 head 上 Approve。cursor[bot] 在同一 head 上也给出了 CHANGES_REQUESTED / DO NOT PASS,结论一致。请在处理完上述两点后(通常会形成新的 head)再提请复核。

@linyueqian

Copy link
Copy Markdown
Contributor

Notes on --decision paired and --require-explanation at 1b2b4593.

The module docstring already says the interval's coverage is not calibrated. Near the 5-pair floor it is off by enough to change what a paired PASS means. At 5, 7 and 13 pairs, bootstrap_median_interval (src/inferrsi/harness/measurement.py L73-L80) returns an order statistic as the lower bound: the smallest pair ratio at 5 pairs, the 2nd smallest at 7 and the 4th smallest at 13. For independent continuous pair ratios whose median equals the gate, PASS then comes with probability P(Bin(n, 1/2) <= j-1): 3.1% at 5 pairs, 6.25% at 7 and 4.6% at 13. The one-sided nominal rate is 2.5%. A 0% gate is a speedup of 1 in both directions, so these are also the A/A rates when the median pair ratio is 1. Your own rates() at 7 repetitions and a 0% gate gives A/A PASS 0.075 (SE 0.008, exact 0.0625) and coverage 0.857 (exact 0.875). The quoted 0.020 holds because the 1% gate sits above no change, and only 5, 10 and 20 pairs ran. A nonrandomized exact sign test keeps the rate at or below 2.5%, and it can first reject at 6 pairs (2^-5 = 3.1%, 2^-6 = 1.6%).

Three smaller points:

  1. A region declared unchanged passes when its interval includes 1 (measurement.py L299-L300). That is not evidence of no change, and there is no multiplicity control. With k such regions and independent noise, a false contradicted comes at 1-(1-q)^k, where q is 6.25% at 5 pairs and 12.5% at 7. That is about 49% for 5 regions at 7 pairs. When the paired decision gives PASS, --require-explanation then turns it into FAIL.
  2. This is separate from the switch-priority question already raised. A straddling interval gives FAIL within_noise (project.py L192-L193), but it does not show that the threshold is missed. docs/contributing.md L138 defines FAIL as evidence that a gate was not met. If FAIL here means "not shown to pass", please say so. Otherwise INCONCLUSIVE would match evidence/qwen38-h20-fp8-single-request-vllm032/validator/statistics-8k-v1/paired_statistics_8k_v1.py L147, which gives INCONCLUSIVE when the point passes and the bound crosses.
  3. Paired records still write report_only: true (measurement.py L134) when the interval decides the verdict.

Both switches are off by default, so this is a good time to fix these.

BeatSeat added a commit to BeatSeat/InferRSI that referenced this pull request Nov 16, 2026
…ithin noise

The percentile bootstrap lower bound for a median of n pairs is an order
statistic whose one-sided error exceeds 2.5% at small n (3.1% at 5 pairs,
6.25% at 7). Use the exact distribution-free order-statistic interval (the
inverted sign test) instead: the rank keeps each tail at or below 2.5%, so the
minimum is six pairs and A/A at the gate passes at most 2.5% of the time.

Declared expectations are judged as a family: region and counter intervals use
the Bonferroni confidence 1 - 5%/k, and too few pairs for it is unverifiable.
A declared-unchanged item is reported as not contradicted, never confirmed.

A paired interval that straddles the gate shows only that passing was not
demonstrated, which is not evidence that the gate was missed, so the round is
INCONCLUSIVE (within_noise) and still does not block the search. Records say
report_only: false when the interval decides the verdict.

Reported by linyueqian on InferMatrixLab#457.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
@BeatSeat
BeatSeat force-pushed the feat/paired-measurement-statistics branch from 1b2b459 to ac0510b Compare November 16, 2026 09:17
@BeatSeat

Copy link
Copy Markdown
Author

已在新 head ac0510b2 上处理三位评审的意见(基于当前 main 016c7c7d)。

amy-why-3459 / cursor[bot] / steward 条件

  1. Rebase:已 rebase 到 016c7c7d(Package shared technical contracts and retire root modules #453、refactor(C2): relocate resource facade and split evidence layout #458、refactor(R1): retire eleven roots and bind Apps inspection #459、Extend VLA observation-to-action benchmark guidance #342、Wire the Harness loop evaluation port through Apps #462、knowledge: reject foreign records in a source-snapshot pack #463 之后),没有冲突;按新 head 重跑了定向 harness 测试和相关 campaign/dashboard 测试,见 PR 正文。
  2. 两开关优先级:--decision 先给出性能结论,--require-explanation 只把 PASS 降为 FAIL 或 INCONCLUSIVE,不改写非 PASS 的结论。已写入 docs/optimization-workflow-unification.md 和 rfc: 可解释的改进——用 profile 证据和配对统计判定 PASS #454。test_both_switches_apply_the_decision_before_the_explanation 覆盖 6 种组合(区间跨门槛 + unverifiable / 未声明、区间低于门槛 + 未支持、配对 PASS + 未声明 / 矛盾 / 已解释)。
  3. 点估计未过、区间过门槛:--decision paired 只看区间,此时判 PASS 并记录 agrees_with_point_gate: false。test_paired_decision_passes_when_only_the_point_estimate_misses_the_gate 用 10 对数据构造点估计 +0.5%(未过 1% 门槛)、区间 [1.07, 1.07] 的情形。
  4. 默认关闭时只报告:原有回归测试保留,report_only: true;新增 test_failing_profile_report_does_not_change_the_default_verdict,profile 报告出错时记为 unavailable,不改判定。

linyueqian(谢谢,复算全部成立)

  1. 区间的单侧误差:改用精确的次序统计量区间(符号检验反演),秩 j 取使 P(Bin(n, 1/2) ≤ j−1) ≤ 2.5% 的最大值,并记录 order_statistic_rank 和 achieved_confidence。MIN_PAIRS 由 5 改为 6,--decision paired 至少要 6 次重复。门槛为 0% 时的 A/A 仿真结果见 PR 正文,每个对数都不超过对应的精确上界;test_interval_rank_is_the_largest_exact_rank_within_two_and_a_half_percent 对 n = 1…40 逐一核对秩的最大性。
  2. "不变"与多重比较:声明 k 条预期时,所有区域和计数使用 Bonferroni 置信度 1 − 5%/k(记录 family_size、confidence、pairs_needed),对数不足时为 unverifiable。每条核对记录 outcome;unchanged 只会是 not_contradicted 或 contradicted,不会被"确认不变"。explain() 拒绝按更少条数校正的证据。
  3. 区间跨过门槛:按 docs/contributing.md 的定义改为 INCONCLUSIVE(within_noise),区间整体低于门槛仍为 FAIL,与 qwen38 验证器的处理一致。这类轮次不被采纳,也不阻塞搜索。
  4. report_only:--decision paired 时为 false。

另外 Codex CLI(gpt-6.1-sol,xhigh)对新 head 做了一轮只读对抗评审,结论和处理见 PR 正文。请在新 head 上复核。

amy-why-3459
amy-why-3459 previously approved these changes Nov 16, 2026

@amy-why-3459 amy-why-3459 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 016c7c7df023944f0f46d8e777d6d31f1df1764c and head ac0510b21b887e34b7357b13dc3d2eb7c9e86c5f(draft)。

上次在 1b2b4593cb72ef7b956f82f000313a9910cffbd7 上的阻塞已处理:

  1. #454 已附条件接受(steward 评论 6897439035)。
  2. 已 rebase 到当前 main 016c7c7d(ahead 8 / behind 0)。
  3. 两开关优先级与「点估计未过、区间过门槛」已写入 docs/optimization-workflow-unification.md 和 #454,并有 test_both_switches_apply_the_decision_before_the_explanation、test_paired_decision_passes_when_only_the_point_estimate_misses_the_gate 覆盖。

本轮还核对了 linyueqian 在旧 head 上的意见及后续修订:精确次序统计量区间、MIN_PAIRS=6、Bonferroni 家族校正、跨门槛记 INCONCLUSIVE(within_noise)、--decision paired 时 report_only: false,以及 profile 报告失败不改变默认判定。我用 n=1…40 的二项尾计算复核了 interval_rank,与文档中的对数要求(k=1 为 6,k=2–3 为 7,k=4–6 为 8,k=7–12 为 9)一致。declarations 已通过。我没有重跑 tools/measurement_simulation.py 的仿真,也没有跑 hosted 全量 pytest。

@david6666666 david6666666 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One issue with how normalized evidence affects the explanation gate.

[v / k for v, k in zip(c, clock[1])],
confidence=confidence)
entry.update(normalized=normalized, observed=normalized["observed"],
basis="normalized_to_reference_regions")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could a sign reversal between raw and normalized evidence stay inconclusive unless the mechanism explicitly predicts a relative effect? This overwrites the raw direction. In a 10-pair check, the raw region interval shows a 20% increase, the normalized interval shows a decrease, and the reference interval merely crosses 1. The result is explained for an absolute decrease. Please preserve that disagreement in the explanation gate and add a regression test.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, agreed and fixed in 6a3c1236. When a region's raw and reference-normalized intervals point in opposite definite directions, the evidence now records observed: "raw_normalized_conflict" and keeps both raw and normalized. That check's outcome is unverifiable, so explanation.status is unverifiable (INCONCLUSIVE under --require-explanation). The normalized sign no longer wins. The mechanism format has no way to declare a relative-only effect, so a reversal is never read as either direction.

A raw no-clear-change with a definite normalized direction is still resolved by normalization; that's the drift it is meant to cancel. A definite raw direction with no clear normalized change stays not_shown.

Regression: test_raw_and_normalized_sign_reversal_is_not_read_as_either_direction reproduces your 10-pair case. The raw target is +20%; two references each cross 1 in different pairs, so neither is contradicted, yet their sum rises in every pair. The normalized result is a decrease. Before the fix this was explained; now it is unverifiable. test_normalization_still_resolves_a_raw_change_hidden_by_drift keeps the intended normalization path. Codex also built a single-reference variant and checked 8,000 direction combinations without finding another reversal.

cursor[bot]
cursor Bot previously approved these changes Nov 16, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 016c7c7 and head ac0510b (draft).

Re-review after author push from prior CHANGES_REQUESTED on 1b2b459. Prior blockers cleared on this head:

  1. Governing #454 acceptance already recorded.
  2. Rebased onto 016c7c7 with focused harness tests reported in the PR body.
  3. Dual-switch priority and point/interval disagreement documented in docs/optimization-workflow-unification.md and covered by test_both_switches_apply_the_decision_before_the_explanation plus test_paired_decision_passes_when_only_the_point_estimate_misses_the_gate (agrees_with_point_gate is False). Defaults-off report-only path retained.

Merge process (not content blockers on this draft head): still draft; head is behind current main again after later merges — rebase before marking ready; run ready-lane CI on the rebased tip. Hardware A/A remains non-blocking while switches default off.

Acting steward re-review while hosted Dong-bot is on usage_limit; formal Approve as cursor[bot].

BeatSeat and others added 9 commits November 16, 2026 21:20
Pair the k-th baseline and candidate samples that steps() already
alternates, and report the median paired speedup with a seeded percentile
bootstrap interval and its relation to the frozen gate. The field is
report-only: verdict, eligibility and selection are unchanged, and any value
that cannot be computed is reported as unavailable so records stay strict
JSON. Fewer than five pairs report no interval because simulated coverage at
three pairs is about 75%. tools/measurement_simulation.py reproduces the
synthetic ablation.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=high] (review)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
…ches

Measurement scripts may add per-step region breakdowns and work counters.
A round may declare its mechanism and expected region/counter directions.
Regions declared unchanged act as an in-step clock, so ratios to them and
their share of the step total cancel device-wide clock or thermal drift;
counters such as launches or DRAM bytes show whether a small gain is
reproducible. Directions come only from paired intervals, with no
repository-wide tolerance.

Two switches are frozen at campaign start and default to the existing
behaviour: --require-explanation adopts only candidates whose mechanism is
confirmed by the data, and --decision paired requires the paired interval
to clear the gate, failing within-noise rounds without blocking the search.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
… evidence

Bind the round's mechanism into control state before the candidate runs and
assess against that copy, so a candidate cannot rewrite its own declaration.
Require five pairs before any profile direction, treat zero-baseline ratios as
undefined, report unavailable shares instead of raising, keep raw evidence for
every region, and never normalize declared references against each other.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
The candidate runs with the caller's privileges and can write state.json or
the attempt copy. Pass the declaration held since submission to the assessor,
record its digest, and treat a copy that no longer matches as an integrity
failure.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
An interrupted attempt is settled by a new process that can only read files
the candidate could write. Settle it without the declaration, mark the record,
and leave require_explanation to keep the round unadopted.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
With both switches on, --decision sets the performance verdict first and
--require-explanation only downgrades a PASS. The paired decision reads the
interval alone, so a candidate whose point estimate misses the gate but whose
paired interval clears it passes, with agrees_with_point_gate recorded false.
Regression tests pin both behaviors.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
…ithin noise

The percentile bootstrap lower bound for a median of n pairs is an order
statistic whose one-sided error exceeds 2.5% at small n (3.1% at 5 pairs,
6.25% at 7). Use the exact distribution-free order-statistic interval (the
inverted sign test) instead: the rank keeps each tail at or below 2.5%, so the
minimum is six pairs and A/A at the gate passes at most 2.5% of the time.

Declared expectations are judged as a family: region and counter intervals use
the Bonferroni confidence 1 - 5%/k, and too few pairs for it is unverifiable.
A declared-unchanged item is reported as not contradicted, never confirmed.

A paired interval that straddles the gate shows only that passing was not
demonstrated, which is not evidence that the gate was missed, so the round is
INCONCLUSIVE (within_noise) and still does not block the search. Records say
report_only: false when the interval decides the verdict.

Reported by linyueqian on InferMatrixLab#457.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
A float 2 ** pairs overflows near a thousand pairs; compare binomial tails as
exact rationals instead. A failure while building profile evidence is reported
as unavailable rather than raised, so it cannot change the default verdict.

Found by Codex review of the exact-interval change.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
When a region's raw and reference-normalized intervals point in opposite
definite directions, an absolute increase can read as a relative decrease
(two references can each cross 1 while their sum rises in every pair). No
declaration predicts a relative effect, so record raw_normalized_conflict and
leave that check unverifiable instead of letting the normalized sign win.

Under --require-explanation, only contradicted evidence fails. An unsupported
prediction is missing evidence, like a paired interval within noise, so it is
INCONCLUSIVE; the candidate is still not adopted and the search continues.

Reported by david6666666 on InferMatrixLab#454 and InferMatrixLab#457.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
@BeatSeat
BeatSeat dismissed stale reviews from cursor[bot] and amy-why-3459 via 6a3c123 November 16, 2026 13:46
@BeatSeat
BeatSeat force-pushed the feat/paired-measurement-statistics branch from ac0510b to 6a3c123 Compare November 16, 2026 13:46
@BeatSeat

Copy link
Copy Markdown
Author

New head 6a3c1236, rebased onto current main 6c7e2379 (including #461's CLI move; the --decision, --require-explanation and --mechanism-file options now live in src/inferrsi/apps/cli/arguments.py), as cursor[bot]'s merge-process note asked.

It also addresses david6666666's two points:

The 34 related test files and ruff pass on this head. A Codex review of the change found no code issues; its only item was syncing the PR body, which is now done. Please re-review on 6a3c1236.

@amy-why-3459 amy-why-3459 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 6c7e237935a22cc5e90e954d4f397376e753e7d8 and head 6a3c123691b0ba6937084837c62a888b9735f59b(draft)。

对照上次已通过的 ac0510b21b887e34b7357b13dc3d2eb7c9e86c5f,本轮增量主要是 6a3c123691b0ba6937084837c62a888b9735f59b,并已 rebase 到 main 6c7e2379(含 #461,--decision / --require-explanation / --mechanism-file 现位于 src/inferrsi/apps/cli/arguments.py)。

已核对 david6666666 的两点:

  1. --require-explanation 下只有 contradicted 记 FAIL,unsupported / unverifiable / not_declared 记 INCONCLUSIVE(src/inferrsi/harness/project.py 与 tests/test_project_explanation.py)。
  2. raw/normalized 方向冲突记为 raw_normalized_conflict,_check_outcome 将其判为 unverifiable;回归测试 test_raw_and_normalized_sign_reversal_is_not_read_as_either_direction 覆盖了所述 10 对情形,test_normalization_still_resolves_a_raw_change_hidden_by_drift 保留了归一化抵消漂移的路径。文档与 #454 已同步。

合并流程(非本 head 内容阻塞):仍为 draft;相对当前 main 2cbfab66b00d75e69be583ebfeabee2b8f83c4ea(#470/#472)落后 2 个提交,但无文件重叠,GitHub 显示可合并无冲突。标记 ready 前请再 rebase。我没有重跑作者声称通过的相关测试和 ruff,也没有跑 hosted 全量 pytest。

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 6c7e237935a22cc5e90e954d4f397376e753e7d8 and head 6a3c123691b0ba6937084837c62a888b9735f59b (draft).
Re-review after author push from prior PASS on ac0510b21b88 (Approves dismissed by rebase). Incremental head addresses david6666666: raw/normalized definite sign reversal records raw_normalized_conflict → check unverifiable; under --require-explanation only contradicted is FAIL and unsupported/unverifiable/not_declared stay INCONCLUSIVE. Prior blockers remain cleared (#454 accepted; dual-switch priority + point/interval disagreement documented/tested). Still draft; behind current main by 2 non-overlapping commits — rebase before ready. Formal Approve as cursor[bot].

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:harness Harness module: profile, A/B loop, quality gates, evidence reports kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants