Repository navigation
Conversation
Skill evaluationPLANNED: not executed; there are no task outcomes yet.
Author verification steps (nonpaid; not yet run)
python tools/build_skill_index.py --source-revision 3671a98ad340fb19e6595d1c8d818c7e9ca2577a --checkThe declared source pin matches this head's canonical library bytes. The command has not run: retain its exit code and full drift/success output. It checks the generated projection, not claim truth or skill efficacy.
Evidence and logs · Retention: 30 days. Plan bindings, budget, frozen exam and advisory detailsAdvisory: obtain missing evidenceRetained evidence availability: planned / unexecuted. No approval or merge authorization is issued.
Model/settings:
Selection/task metadata (not a relevance verdict): {"adapter":"audit","basis":"trusted_path_mapping","matched_paths":["src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md","src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/metadata.json","src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/gpu-measurement-validity.md"],"tasks":[{"evidence_scope":"http_fixed_workload","id":"audit-threshold","request":"Audit the fixed input report using its contract. Compute the median of per-trial request means, including every request in each trial, and the percentage reduction relative to the baseline aggregate.","split":"holdout"}]}Automatic execution / cumulative PR budget: {"reason":"Trusted configuration or pricing missing, invalid or expired","remaining_usd":"unknown","reserved_or_unresolved_usd":"unknown","spent_or_legacy_usd":"unknown","state":"blocked"}Projection check selection (not an executed result): {"declared_in":"src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json","source_revision":"3671a98ad340fb19e6595d1c8d818c7e9ca2577a","status":"ready"}Provider/settings: Uncovered / unsupported paths (60)
|
|
Thanks for the careful caveats and the two-step generation. I reviewed this against current
I also checked: the new reference is 175 lines; its SHA-256 I am not approving this head until the matrix placement is fixed. |
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base cd8fcb452bd2c7969a7a6e2544aa1cb5f8ad5332 and head 8c0b894a2dc9b8572700ca3f80b119e75f2377db (draft).
Acting reviewer: cursor[bot] via standing InferRSI steward delegation. Binding formal review to current head.
Blockers before Approve / ready-for-review:
- Matrix placement. The new
docs/reference-matrix.mdrow for GPU measurement validity sits under## Task-specific benchmark selection (issue #60)(around L444–L451). The matching searchspace note is under the issue #60 heading as well. This is general GPU measurement methodology, not an #60 task-selection source. Move the matrix row and searchspace paragraph to a dedicated short section (same pattern as## Faster CosyVoice evaluation source context), then regenerate coverage/records sodocs/skill-coverage.mddoes not label it as an #60 source. - Rebase onto current main. Head diverges: ahead 2 / behind 1 vs
cd8fcb45(#453). Rebase, and if the source-revision commit changes, regenerate projections and update the declarationhead_sha. - Governing issue still Proposed. #455 has no recorded owner acceptance yet. Steward decision will be posted on #455; keep this draft until acceptance + the matrix/rebase fixes land.
Checked / not blocking alone: docs-only slice; candidate / empirical not_run preserved; Triton pin and do_bench flush description match prior notes; installed-count 923→924 is coherent with a single new reference. I did not re-download NVIDIA/PyTorch docs or re-run the installed-wheel check.
Re-request review after the blockers are addressed on a new head.
|
I re-checked after the steward decision on #455: the documentation-only slice is now Accepted (with conditions), and the goal-acceptance owner is Dong1017. That external blocker is cleared. This note is for the same head
The “require an explanation before adoption” wording should stay clearly candidate guidance, as accepted on #455. I am not approving this head. cursor[bot] also left CHANGES_REQUESTED / DO NOT PASS on the same head for these items. Please request a re-review once they are addressed (likely on a new head). |
8c0b894 to
6ec2078
Compare
|
Addressed on the new head
|
amy-why-3459
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 016c7c7df023944f0f46d8e777d6d31f1df1764c and head 6ec207892ed4204b9e14d1fc6166681e048b301e (draft).
The previous blockers on 8c0b894a2dc9b8572700ca3f80b119e75f2377db are resolved:
- The
docs/reference-matrix.mdrow and thedocs/searchspace-matrix.mdnote now sit in a dedicated "GPU measurement validity and profiler attribution" section, anddocs/skill-coverage.mdlabels the source under that section rather than issue #60. - The branch is rebased onto current main
016c7c7d(ahead 2 / behind 0). Provenancesource_revisionis3a6dc01c1139d1d97fad52b99cc39be61cad520d, the regeneration commit is this head, and the installed-resource count is 928 (927 on main plus the new reference). - #455 remains accepted with conditions (steward comment 6897438871). Candidate / empirical
not_runwording is preserved; the explanation-first adoption language stays author-proposed guidance.
declarations and knowledge-model-policy passed on this head. I did not re-download the NVIDIA/PyTorch documents to re-check retained hashes, and I did not rerun the generators or the installed-wheel check.
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 016c7c7 and head 6ec2078 (draft).
Re-review after author push from prior CHANGES_REQUESTED on 8c0b894. Prior blockers cleared on this head:
- Matrix row and searchspace note now live under dedicated "GPU measurement validity and profiler attribution" sections (not issue #60); skill-coverage labels match.
- Branch was rebased onto 016c7c7 with regenerated records / updated declaration head_sha.
- Governing #455 acceptance recorded earlier.
Scope: documentation/knowledge slice only; candidate/empirical not_run preserved. Declarations check green.
Merge process (not content blockers on this draft head): still draft; head is behind current main again after later merges — rebase before marking ready; run ready-lane CI on the rebased tip. skill-eval plan failure on this head is setuptools_scm tag-parse infra (evidence-minicpm-… tag), not a content defect in this slice.
Acting steward re-review while hosted Dong-bot is on usage_limit; formal Approve as cursor[bot].
…ance Extend inference-benchmark-design with an owner-local reference on clock and cache controls, within-run attribution, work-count signatures and timing traps. Candidate guidance; empirical validation not run. Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (author) Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
Records generated by tools/build_skill_index.py and tools/build_skill_coverage.py from source revision ee8c924. Agent-assisted-by: Claude Code / claude-opus-5-5 (ran generators) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
…s reference main 6c7e237 (InferMatrixLab#382) added one packaged knowledge resource without updating the installed-resource assertion, so main installs 928 files against an expected 927. With the measurement-validity reference the installed count is 929, verified on a wheel built from a clean archive of this head. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
4458a05
6ec2078 to
4458a05
Compare
|
Thanks for the approvals. Following cursor[bot]'s merge-process note, I rebased onto current main
The |
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 6c7e237935a22cc5e90e954d4f397376e753e7d8 and head 4458a0530a0e2a312eb6ce98af190afe5091b97e (re-review after author rebase; prior Approves on 6ec207892ed4 dismissed).
Verified #455 docs-only slice: gpu-measurement-validity.md retains unverified/source-derived status and no numeric gate; matrix rows in dedicated sections (not #60); skill pointer + regenerated pack/projections; installed count 929 matches +1 reference vs post-#382 base. Still draft and CONFLICTING vs current main after #470/#472 (check_installed 928→ needs rebase keeping 929). Mark ready only after rebase onto current main + ready CI.
Standing steward review delegation while hosted Dong-bot is on usage_limit.
amy-why-3459
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 6c7e237935a22cc5e90e954d4f397376e753e7d8 and head 4458a0530a0e2a312eb6ce98af190afe5091b97e (draft), incremental from previously reviewed 6ec207892ed4204b9e14d1fc6166681e048b301e.
Canonical content is unchanged: gpu-measurement-validity.md, the skill pointer in SKILL.md, and metadata.json are byte-identical to the previously approved head. Matrix rows remain in the dedicated "GPU measurement validity and profiler attribution" section, and docs/skill-coverage.md labels that source under that section (16 readsets; candidate / not_run preserved). Provenance source_revision is ee8c924f61f0a8f17338b8b25909a410398067ad; the installed-resource count is 929 on this head (928 actual files on base 6c7e2379 plus the new reference).
Merge process only (not a content blocker on this draft head): main has since moved to 2cbfab66 (#470, then #472, which sets the installed count to 928), and this branch is CONFLICTING. Rebase before marking ready; the fixture should then become 928→929.
I did not re-download the NVIDIA/PyTorch documents to re-check retained hashes, and I did not rerun the generators or the installed-wheel check.
Governing Issue and delivery slice
Refs #455
GPU E2E timings can move with clock, thermal, power and sharing conditions; profiler replay can introduce different execution conditions. The reference gives a procedure for retaining paired timings, checking conditional within-run attribution and Amdahl predictions, and connecting gains to work counters and a predicted profiler signature. It covers timing/cache/baseline/JIT pitfalls and keeps the four supplied numerical observations explicitly unverified.
The existing
inference-benchmark-designowner already defines comparable workloads, measurement integrity and evidence interpretation. Under the knowledge-base ownership contract, this is an owner-local reference extending that procedure; it does not need a separate skill identity or registry entry. Candidate state, empiricalnot_runand efficacynot_evaluatedremain explicit.Changes since review
Rebased onto
main6c7e2379(after #453, #458, #459, #342, #462, #463, #461, #465, #435, #382 and two direct commits) and regenerated from the new source commit.docs/reference-matrix.mdrow and thedocs/searchspace-matrix.mdnote now sit in dedicatedGPU measurement validity and profiler attributionsections, outside the issue [RFC] Benchmark selection by task for InferRSI #60 section. The regenerateddocs/skill-coverage.mdlabels the source accordingly.metadata.json,reference-matrix.mdandtests/check_installed.pywere resolved by keeping both additions. After the second rebase only the generated files conflicted; they were regenerated from the new source commit.6c7e2379(feat: refine vLLM Ascend ACLGraph and model adaptation knowledge (Refs #350) #382) added one packaged knowledge resource without updatingtests/check_installed.py, so a wheel built from main installs 928 files against the asserted 927. With this reference the installed count is 929, set in a separate commit (4458a053) outside the protected paths and verified on a wheel built from a clean archive of this head.Knowledge authorship declaration
Changed artifacts and evidence
methods/inference-benchmark-design/references/gpu-measurement-validity.md(175 lines); SKILL.md adds six lines_provenance.json, coverage view;tests/check_installed.pycount 929Canonical skill paths above are relative to
src/inferrsi/knowledge/data/skill_library/. The registry, unrelated skill records and generated navigation README remain unchanged.Checks and dependencies
All Python commands used the requester-specified interpreter and checkout
PYTHONPATH. Final Git-dependent checks used a temporary Git 2.49.1 build, with the repository tools unchanged and native--no-lazy-fetchsupport.tools/build_skill_index.py --source-revision ee8c924f61f0a8f17338b8b25909a410398067ad, then--check: passed (exit 0), 80 records current; generated churn limited to the target record,_provenance.jsonand the coverage view.tools/build_skill_coverage.py, then--check: passed (exit 0). The coverage view now labels the source underGPU measurement validity and profiler attribution, not issue [RFC] Benchmark selection by task for InferRSI #60.python -m pytest tests/test_skill_catalog.py tests/test_skill_index.py tests/test_skill_library.py tests/test_skill_library_readers.py tests/test_skill_coverage_view.py tests/test_knowledge_*.py --basetemp <temporary-test-dir>: 485 passed, 5 skipped, 112 subtests passed at the final head.python -m ruff check src tests: passed.git archiveof the final head (pip build isolation;SETUPTOOLS_SCM_PRETEND_VERSION=0.0.0, because theevidence-minicpm-seedtts-dec7a734-v1tag on main breaks setuptools-scm version parsing), installed it noneditably to a temporary target, and rantests/check_installed.pyoutside the checkout with isolated imports: passed, 929 resource files.tools/knowledge_declaration_paths.py --base origin/main --head HEAD: passed (exit 0).assess()fromtools/check_knowledge_model_policy.pyagainst this PR body, final head, base-relative paths and the trustedmainpolicy: compliant, declaration compliance only (7 protected paths).git diff --check, clean-worktree check, DCO sign-off andAgent-assisted-bytrailer inspection on both commits: passed.Resolved validation failures: System Git 2.43 lacks
--no-lazy-fetch; generation and Git-dependent tests used Git 2.49.1. A first wheel built from the working tree picked up a stale localbuild/directory containing retired root modules; the wheel was rebuilt from a cleangit archiveand the installed check passed.Skipped / not run: One browser test requires unavailable Playwright; two fresh-venv tests remain opt-in; two NTFS-junction cases are Windows-only. The separate noneditable wheel check above ran successfully. Full ready/nightly CI, independent review, GPU profiling, model quality, L1–L4 campaign execution and efficacy evaluation were not run.
Limits
ee8c924f, the pinnedsource_revision), then records are regenerated from that commit and committed (f89f7667). A third commit (4458a053) only updates the installed-resource count intests/check_installed.py, which is not a canonical or protected path. The source revision is an ancestor of the head, so no separate bundle is needed.Remaining work and Issue closure