Repository navigation
Conversation
|
我对照当前 main 先说已经对上我在 #454 里提的几条: 还需要处理的点:
我没有重新跑 我不会在这个 head 上 Approve,直到 #454 被接受,并且上面的 rebase / 文档与测试缺口处理完毕。 |
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base cd8fcb452bd2c7969a7a6e2544aa1cb5f8ad5332 and head 1b2b4593cb72ef7b956f82f000313a9910cffbd7 (draft).
Acting reviewer: cursor[bot] via standing InferRSI steward delegation. Binding formal review to current head.
Blockers before Approve / ready-for-review:
- Governing issue not accepted. #454 is still Proposed with goal-acceptance owner TBD. Steward decision will be posted on #454; do not treat this draft as authorized delivery until that record exists.
- Rebase onto current main. Head diverges: ahead 5 / behind 1 vs
cd8fcb45(#453), which also touchedsrc/inferrsi/arguments.py. Rebase and re-run the focused harness tests on the new head; refresh declarationhead_shaif needed. - Dual-switch priority and point/interval disagreement are under-specified in docs/tests. In
assess()(project.py),--decisionruns first;--require-explanationonly rewrites aPASS. So with both switches on,within_noiseFAIL stays FAIL even if explanation isunverifiable. Also, paired mode keys only ongate_relation(above_gate→ PASS), so a point estimate below the gate can still PASS when the interval clears the gate. Document both behaviors in #454 anddocs/optimization-workflow-unification.md, and add regression tests (both switches on;agrees_with_point_gate is Falsewithabove_gate).
Checked / helpful but not enough to Approve: mechanical explanation.status definitions look consistent with the RFC; defaults-off report-only path is covered; mechanism binding / tamper / recover handling is present; tools/measurement_simulation.py exists with stated seeds. I did not re-run the 200×10000 simulation or GPU/NPU hardware checks. Hosted CI on this draft is limited (declarations green); full ready-lane evidence is still missing.
Hardware A/A + known-gain confirmation is not a merge prerequisite while both switches default off (steward decision on #454). Re-request review after acceptance + rebase + the doc/test gaps above.
|
回看了 Dong1017 在 #454 的 steward 决定:三个 owner 决策已经给出(可选能力、默认关闭、按一个 PR 评审),goal-acceptance owner 记为 Dong1017,硬件 A/A 也不再作为合入前提。这一条外部阻塞已经解除。 当前 head 仍是
因此我不会在这个 head 上 Approve。cursor[bot] 在同一 head 上也给出了 CHANGES_REQUESTED / DO NOT PASS,结论一致。请在处理完上述两点后(通常会形成新的 head)再提请复核。 |
|
Notes on The module docstring already says the interval's coverage is not calibrated. Near the 5-pair floor it is off by enough to change what a paired PASS means. At 5, 7 and 13 pairs, Three smaller points:
Both switches are off by default, so this is a good time to fix these. |
…ithin noise The percentile bootstrap lower bound for a median of n pairs is an order statistic whose one-sided error exceeds 2.5% at small n (3.1% at 5 pairs, 6.25% at 7). Use the exact distribution-free order-statistic interval (the inverted sign test) instead: the rank keeps each tail at or below 2.5%, so the minimum is six pairs and A/A at the gate passes at most 2.5% of the time. Declared expectations are judged as a family: region and counter intervals use the Bonferroni confidence 1 - 5%/k, and too few pairs for it is unverifiable. A declared-unchanged item is reported as not contradicted, never confirmed. A paired interval that straddles the gate shows only that passing was not demonstrated, which is not evidence that the gate was missed, so the round is INCONCLUSIVE (within_noise) and still does not block the search. Records say report_only: false when the interval decides the verdict. Reported by linyueqian on InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
1b2b459 to
ac0510b
Compare
|
已在新 head amy-why-3459 / cursor[bot] / steward 条件
linyueqian(谢谢,复算全部成立)
另外 Codex CLI(gpt-6.1-sol,xhigh)对新 head 做了一轮只读对抗评审,结论和处理见 PR 正文。请在新 head 上复核。 |
amy-why-3459
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 016c7c7df023944f0f46d8e777d6d31f1df1764c and head ac0510b21b887e34b7357b13dc3d2eb7c9e86c5f(draft)。
上次在 1b2b4593cb72ef7b956f82f000313a9910cffbd7 上的阻塞已处理:
- #454 已附条件接受(steward 评论 6897439035)。
- 已 rebase 到当前 main
016c7c7d(ahead 8 / behind 0)。 - 两开关优先级与「点估计未过、区间过门槛」已写入
docs/optimization-workflow-unification.md和 #454,并有test_both_switches_apply_the_decision_before_the_explanation、test_paired_decision_passes_when_only_the_point_estimate_misses_the_gate覆盖。
本轮还核对了 linyueqian 在旧 head 上的意见及后续修订:精确次序统计量区间、MIN_PAIRS=6、Bonferroni 家族校正、跨门槛记 INCONCLUSIVE(within_noise)、--decision paired 时 report_only: false,以及 profile 报告失败不改变默认判定。我用 n=1…40 的二项尾计算复核了 interval_rank,与文档中的对数要求(k=1 为 6,k=2–3 为 7,k=4–6 为 8,k=7–12 为 9)一致。declarations 已通过。我没有重跑 tools/measurement_simulation.py 的仿真,也没有跑 hosted 全量 pytest。
david6666666
left a comment
There was a problem hiding this comment.
One issue with how normalized evidence affects the explanation gate.
| [v / k for v, k in zip(c, clock[1])], | ||
| confidence=confidence) | ||
| entry.update(normalized=normalized, observed=normalized["observed"], | ||
| basis="normalized_to_reference_regions") |
There was a problem hiding this comment.
Could a sign reversal between raw and normalized evidence stay inconclusive unless the mechanism explicitly predicts a relative effect? This overwrites the raw direction. In a 10-pair check, the raw region interval shows a 20% increase, the normalized interval shows a decrease, and the reference interval merely crosses 1. The result is explained for an absolute decrease. Please preserve that disagreement in the explanation gate and add a regression test.
There was a problem hiding this comment.
Thanks, agreed and fixed in 6a3c1236. When a region's raw and reference-normalized intervals point in opposite definite directions, the evidence now records observed: "raw_normalized_conflict" and keeps both raw and normalized. That check's outcome is unverifiable, so explanation.status is unverifiable (INCONCLUSIVE under --require-explanation). The normalized sign no longer wins. The mechanism format has no way to declare a relative-only effect, so a reversal is never read as either direction.
A raw no-clear-change with a definite normalized direction is still resolved by normalization; that's the drift it is meant to cancel. A definite raw direction with no clear normalized change stays not_shown.
Regression: test_raw_and_normalized_sign_reversal_is_not_read_as_either_direction reproduces your 10-pair case. The raw target is +20%; two references each cross 1 in different pairs, so neither is contradicted, yet their sum rises in every pair. The normalized result is a decrease. Before the fix this was explained; now it is unverifiable. test_normalization_still_resolves_a_raw_change_hidden_by_drift keeps the intended normalization path. Codex also built a single-reference variant and checked 8,000 direction combinations without finding another reversal.
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 016c7c7 and head ac0510b (draft).
Re-review after author push from prior CHANGES_REQUESTED on 1b2b459. Prior blockers cleared on this head:
- Governing #454 acceptance already recorded.
- Rebased onto 016c7c7 with focused harness tests reported in the PR body.
- Dual-switch priority and point/interval disagreement documented in docs/optimization-workflow-unification.md and covered by test_both_switches_apply_the_decision_before_the_explanation plus test_paired_decision_passes_when_only_the_point_estimate_misses_the_gate (agrees_with_point_gate is False). Defaults-off report-only path retained.
Merge process (not content blockers on this draft head): still draft; head is behind current main again after later merges — rebase before marking ready; run ready-lane CI on the rebased tip. Hardware A/A remains non-blocking while switches default off.
Acting steward re-review while hosted Dong-bot is on usage_limit; formal Approve as cursor[bot].
Pair the k-th baseline and candidate samples that steps() already alternates, and report the median paired speedup with a seeded percentile bootstrap interval and its relation to the frozen gate. The field is report-only: verdict, eligibility and selection are unchanged, and any value that cannot be computed is reported as unavailable so records stay strict JSON. Fewer than five pairs report no interval because simulated coverage at three pairs is about 75%. tools/measurement_simulation.py reproduces the synthetic ablation. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=high] (review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
…ches Measurement scripts may add per-step region breakdowns and work counters. A round may declare its mechanism and expected region/counter directions. Regions declared unchanged act as an in-step clock, so ratios to them and their share of the step total cancel device-wide clock or thermal drift; counters such as launches or DRAM bytes show whether a small gain is reproducible. Directions come only from paired intervals, with no repository-wide tolerance. Two switches are frozen at campaign start and default to the existing behaviour: --require-explanation adopts only candidates whose mechanism is confirmed by the data, and --decision paired requires the paired interval to clear the gate, failing within-noise rounds without blocking the search. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
… evidence Bind the round's mechanism into control state before the candidate runs and assess against that copy, so a candidate cannot rewrite its own declaration. Require five pairs before any profile direction, treat zero-baseline ratios as undefined, report unavailable shares instead of raising, keep raw evidence for every region, and never normalize declared references against each other. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
The candidate runs with the caller's privileges and can write state.json or the attempt copy. Pass the declaration held since submission to the assessor, record its digest, and treat a copy that no longer matches as an integrity failure. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
An interrupted attempt is settled by a new process that can only read files the candidate could write. Settle it without the declaration, mark the record, and leave require_explanation to keep the round unadopted. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
With both switches on, --decision sets the performance verdict first and --require-explanation only downgrades a PASS. The paired decision reads the interval alone, so a candidate whose point estimate misses the gate but whose paired interval clears it passes, with agrees_with_point_gate recorded false. Regression tests pin both behaviors. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
…ithin noise The percentile bootstrap lower bound for a median of n pairs is an order statistic whose one-sided error exceeds 2.5% at small n (3.1% at 5 pairs, 6.25% at 7). Use the exact distribution-free order-statistic interval (the inverted sign test) instead: the rank keeps each tail at or below 2.5%, so the minimum is six pairs and A/A at the gate passes at most 2.5% of the time. Declared expectations are judged as a family: region and counter intervals use the Bonferroni confidence 1 - 5%/k, and too few pairs for it is unverifiable. A declared-unchanged item is reported as not contradicted, never confirmed. A paired interval that straddles the gate shows only that passing was not demonstrated, which is not evidence that the gate was missed, so the round is INCONCLUSIVE (within_noise) and still does not block the search. Records say report_only: false when the interval decides the verdict. Reported by linyueqian on InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
A float 2 ** pairs overflows near a thousand pairs; compare binomial tails as exact rationals instead. A failure while building profile evidence is reported as unavailable rather than raised, so it cannot change the default verdict. Found by Codex review of the exact-interval change. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
When a region's raw and reference-normalized intervals point in opposite definite directions, an absolute increase can read as a relative decrease (two references can each cross 1 while their sum rises in every pair). No declaration predicts a relative effect, so record raw_normalized_conflict and leave that check unverifiable instead of letting the normalized sign win. Under --require-explanation, only contradicted evidence fails. An unsupported prediction is missing evidence, like a paired interval within noise, so it is INCONCLUSIVE; the candidate is still not adopted and the search continues. Reported by david6666666 on InferMatrixLab#454 and InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
6a3c123
ac0510b to
6a3c123
Compare
|
New head It also addresses david6666666's two points:
The 34 related test files and ruff pass on this head. A Codex review of the change found no code issues; its only item was syncing the PR body, which is now done. Please re-review on |
amy-why-3459
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 6c7e237935a22cc5e90e954d4f397376e753e7d8 and head 6a3c123691b0ba6937084837c62a888b9735f59b(draft)。
对照上次已通过的 ac0510b21b887e34b7357b13dc3d2eb7c9e86c5f,本轮增量主要是 6a3c123691b0ba6937084837c62a888b9735f59b,并已 rebase 到 main 6c7e2379(含 #461,--decision / --require-explanation / --mechanism-file 现位于 src/inferrsi/apps/cli/arguments.py)。
已核对 david6666666 的两点:
--require-explanation下只有contradicted记 FAIL,unsupported/unverifiable/not_declared记 INCONCLUSIVE(src/inferrsi/harness/project.py与tests/test_project_explanation.py)。- raw/normalized 方向冲突记为
raw_normalized_conflict,_check_outcome将其判为unverifiable;回归测试test_raw_and_normalized_sign_reversal_is_not_read_as_either_direction覆盖了所述 10 对情形,test_normalization_still_resolves_a_raw_change_hidden_by_drift保留了归一化抵消漂移的路径。文档与 #454 已同步。
合并流程(非本 head 内容阻塞):仍为 draft;相对当前 main 2cbfab66b00d75e69be583ebfeabee2b8f83c4ea(#470/#472)落后 2 个提交,但无文件重叠,GitHub 显示可合并无冲突。标记 ready 前请再 rebase。我没有重跑作者声称通过的相关测试和 ruff,也没有跑 hosted 全量 pytest。
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 6c7e237935a22cc5e90e954d4f397376e753e7d8 and head 6a3c123691b0ba6937084837c62a888b9735f59b (draft).
Re-review after author push from prior PASS on ac0510b21b88 (Approves dismissed by rebase). Incremental head addresses david6666666: raw/normalized definite sign reversal records raw_normalized_conflict → check unverifiable; under --require-explanation only contradicted is FAIL and unsupported/unverifiable/not_declared stay INCONCLUSIVE. Prior blockers remain cleared (#454 accepted; dual-switch priority + point/interval disagreement documented/tested). Still draft; behind current main by 2 non-overlapping commits — rebase before ready. Formal Approve as cursor[bot].
Governing Issue and delivery slice
Refs #454
campaign start/round/final)新增三项能力,默认都不改变现有判定:steps()已有的 AB/BA 交替,把同一次重复里的基线和候选样本配对,报告逐对加速比中位数的精确次序统计量区间(符号检验反演,至少 6 对),以及区间与门槛的关系。breakdown(分区域耗时)和counters(工作量计数);每轮可以用--mechanism-file声明机制和预期方向;Harness 按声明条数做 Bonferroni 校正后机械核对,给出每条核对的outcome和explanation.status。--require-explanation只采纳explained的候选;--decision paired用配对区间判定,区间跨门槛判 INCONCLUSIVE(within_noise),不被采纳也不阻塞搜索。两者同时打开时,先由--decision得出性能结论,--require-explanation只降级 PASS。benchmarks session拒绝--mechanism-file);Dashboard 展示;与具体 profiler 绑定的采集脚本;GPU/NPU 实测。Knowledge authorship declaration
本 PR 不触及 knowledge-model policy 的受保护路径,预期为 Not applicable。
Changed artifacts and evidence
src/inferrsi/harness/measurement.pyinterval_rank、median_interval、min_pairs,二项尾用有理数精确比较)、配对统计、profile_evidence(Bonferroni)(原始与归一化方向相反时记为raw_normalized_conflict)、validate_mechanism、explain;无法计算的值一律返回unavailable,记录保持严格 JSONsrc/inferrsi/harness/project.pyassess()接入两个开关和机制核对;profile 报告出错时记为unavailable,不影响判定;新增设置项的校验src/inferrsi/harness/optimization.pysrc/inferrsi/harness/optimization_session.py--mechanism-file;_runtime_identity纳入measurement.pysrc/inferrsi/apps/cli/arguments.py--decision、--require-explanation、--mechanism-filedocs/optimization-workflow-unification.mdtests/test_harness_measurement.pyunchanged只记"未推翻"、原始与归一化方向相反不被读成任一方向、机制校验与分类tests/test_project_explanation.pywithin_noise为 INCONCLUSIVE;两开关组合 6 例;点估计未过而区间过门槛判 PASS;profile 报告出错不改判;篡改state.json或副本;中断后伪造声明的恢复tests/test_project_optimization.py、tests/test_optimization_session.pytools/measurement_simulation.py--gate-pct 0给出门槛处 A/A 比例与精确上界判定逻辑
explanation.status--require-explanation下explainedconfirmed,且没有任何一条contradictedunsupportednot_shown(未检出;区间仍可能包含预期的变化)contradictedunchanged的项检出变化unverifiablenot_declared--decision paired区间与门槛above_gatepaired_interval_above_gate(即使点估计未过门槛;此时agrees_with_point_gate: false)straddles_gatewithin_noise(没能证明过门槛,不是门槛未达到的证据)below_gate、gate_unreachableperformance_gate_not_metpaired_statistics_unavailable,不退回未配对规则在
--require-explanation下只有contradicted(数据与预期相反)记 FAIL,证据不足的情形都记 INCONCLUSIVE,与配对性能判定的原则一致。若某区域原始比较和"除以参照区域"后的比较给出明确相反的方向(例如原始增加 20%、相对参照却下降),记为raw_normalized_conflict,该条核对为unverifiable:机制声明无法表达"只预期相对变化",两个方向都不采信。声明为
unchanged的项只会被记为not_contradicted(未检出变化)或contradicted,不会被"确认不变":没有等价性容差时,区间包含 1 不是"没变"的证据。声明为unchanged的区域同时是本轮的假设:其他区域会另外给出"除以同一步骤参照区域"后的比较,share_based_speedup用参照占比推算整体加速。只有让一个步骤内所有区域按同一比例缩放的变化,才会被这样抵消。参照区域本身只做原始比较,不会互相归一化。合成消融(
python tools/measurement_simulation.py,CPU 仿真,非设备实测)每格 200 次 campaign,数据 seed 20261115;不配对区间这一对照规则用 bootstrap 10000 次。200 次 campaign 的 Monte Carlo 标准误约 0.01–0.03,门槛处的误报以下方精确上界和大样本复核为准。
Checks and dependencies
ruff check src tests tools:通过。6c7e2379)上运行 34 个 harness / campaign / project / dashboard / optimization 相关测试文件(含tests/test_harness_measurement.py、tests/test_project_explanation.py、tests/test_project_optimization.py、tests/test_optimization_session.py、tests/test_module_boundaries.py和 Dashboard 测试):全部通过,无失败。Git 相关测试使用临时 Git 2.49.1(本机 2.43 不支持--no-lazy-fetch)。gpt-6.1-sol,xhigh)本地只读评审,每一轮都用对抗输入去复现问题。Infinity;零基线计数在 1 对时就给出确定方向;breakdown 全为零时抛异常;多个参照区域互相归一化;候选在执行期间改写mechanism.json或state.json中的声明;中断后经recover接受伪造的声明(现记录mechanism_unverified_after_recovery,按未声明处理)。2 ** n转浮点溢出,且 profile 报告异常会改变默认判定。已修复(有理数精确比较;报告出错记为unavailable)并补回归测试。第八轮复核结论:Ready(1024 对复现通过;注入报告异常后默认判定不变;31,255 组精确秩检查无偏差)。unverifiable;8,000 组方向组合检查无反向误判;原始样本不足或零基线不会经归一化变成明确方向;assess 与独立 final 都是unsupported → INCONCLUSIVE、contradicted → FAIL,搜索继续;rebase 后无旧 CLI 路径残留。结论 Ready with fixes,唯一一项是同步本 PR 说明(即本次更新)。plan.json已有plan_digest保护。measurement.py只用标准库。steps()两侧等长且交替执行;区间的精确性依赖各次重复独立、比值来自连续分布,离散计数出现并列时偏保守。harness/*.py会改变 runtime identity,旧 runtime 启动的会话需要用原来冻结的 runtime 恢复(既有规则)。评审后的修订
6c7e2379(含 #461 的 CLI 迁移),无冲突;新 head 上重跑测试test_both_switches_apply_the_decision_before_the_explanation(6 例)、test_paired_decision_passes_when_only_the_point_estimate_misses_the_gateorder_statistic_rank、achieved_confidenceunchanged不是证据,且没有多重比较校正family_size、confidence、pairs_needed);unchanged只记not_contradicted/contradicteddocs/contributing.md的 FAIL 定义冲突within_noise);低于门槛仍为 FAILreport_only: true--decision paired时为falseraw_normalized_conflict,该条核对unverifiable;test_raw_and_normalized_sign_reversal_is_not_read_as_either_direction复现了他描述的 10 对情形(两个参照各自跨过 1,但总和每对都上升),修复前判explained,修复后为unverifiableunsupported是证据不足,应与contradicted区分--require-explanation下unsupported改为 INCONCLUSIVE,只有contradicted记 FAIL;候选都不被采纳Remaining work and Issue closure
Remaining Issue requirements / follow-up:
hypothesis、verdict 和采纳结果;mechanism、explanation(含每条outcome)、profile_evidence、paired_statistics已在同一份控制记录里,后续可在 Apps 视图层只读投影,不另写记录;另有互补的知识 PR(task: add GPU measurement-validity and profiler-attribution guidance to inference-benchmark-design #455),两者互不依赖。
Goal-acceptance owner or accepted closure delegation: Dong1017(rfc: 可解释的改进——用 profile 证据和配对统计判定 PASS #454 记录);未授予关闭委托。
Goal ready for acceptance? Not claimed