diff --git a/docs/reference-matrix.md b/docs/reference-matrix.md
index c51090aee..fcca4bc13 100644
--- a/docs/reference-matrix.md
+++ b/docs/reference-matrix.md
@@ -451,6 +451,14 @@ the candidate mechanisms.
| [yuekaizhang/faster-cosyvoice @ f1049c59](https://github.com/yuekaizhang/faster-cosyvoice/tree/f1049c59196c2c1a890079abc56056b43f27cd1d), inspected 2026-11-03; overall source rights UNKNOWN, selected file headers retain Apache-2.0 and unresolved inherited attribution | Bounded PCM client/server-credit, chunk planner, session/worker, Token2Wav and timing/quality-report inspection; [exact readset and per-file rights](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/faster-cosyvoice-source.json) | Extend [inference-benchmark-design](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md) with [dependency-aware TTS playout](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/dependency-aware-tts-playout.md): route-origin credit versus client zero-buffer playback, same-session Mel dependencies, cumulative vocoding and partial-cost/token-derived denominators. Original analysis with immutable links only; candidate / empirical not_run / efficacy not_evaluated. Author reports remain separate; no copied implementation/media, new runtime or hardware/quality/service qualification. |
| [Embodied.cpp repo](https://github.com/SEU-PAISys/Embodied.cpp/tree/156bed151826780e2ce0132f8d0e3540889ae8cd) + [paper v3](https://arxiv.org/abs/2607.02501v3), inspected 2026-11-04 (repository Apache-2.0; paper CC BY 4.0) | Selected fixed CPU-observation-to-complete-CPU-action boundary, client queue/profiler denominators, precision identity, per-task quality and source/artifact lineage; [exact readset and rights](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/embodied-cpp-source.json). Later SmolVLA reports remain separate from the August paper. | Extend registered owner `inference-benchmark-design` ([SKILL.md](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md)) with [VLA boundary and quality guidance](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/embodied-cpp-boundary-and-quality.md). Source-derived candidate/not_run; public sample arithmetic is distinct from episode recount and native execution. Generated actions are not control frequency; per-task loss, unequal precision, whole-device memory and unresolved measured-script/dirty-patch identity remain explicit. |
+## GPU measurement validity and profiler attribution
+
+General GPU measurement methodology for `inference-benchmark-design`, not an issue #60 task-selection source.
+
+| Source / revision | Inspected scope | Reuse, extraction and limits |
+| --- | --- | --- |
+| NVIDIA NCU 2025.3, Nsight Systems 2025.3, CUDA 12.9.1/10.2 archives, hash-pinned SMI document; Triton v3.2.0 @ `9641643da6c52000c807b5eeed05edaec4402a67`; PyTorch 2.7; [robust-kbench paper v1](https://arxiv.org/html/2509.14279v1); [typed source inventory](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/metadata.json) | Selected clock/cache/replay, timing/graph, metric-name, flush, extension-identity and benchmark-exploit contracts, inspected 2026-11-15 | Extend `inference-benchmark-design` with [GPU measurement validity and profiler attribution](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/gpu-measurement-validity.md). Candidate / empirical not_run / efficacy not_evaluated; normalization/adoption is an unverified author proposal, four task-supplied cases lack raw evidence, and source review does not establish a gain or current-device metric availability. |
+
## SGLang versioned entry selection
The [three-pin responsibility comparison](../src/inferrsi/knowledge/data/skill_library/repositories/sglang/SKILL.md#choose-among-source-pins)
diff --git a/docs/searchspace-matrix.md b/docs/searchspace-matrix.md
index 1c4ccc1fb..916c92920 100644
--- a/docs/searchspace-matrix.md
+++ b/docs/searchspace-matrix.md
@@ -654,6 +654,16 @@ remains unchanged; source use of GPU, TensorRT, Flow or HiFT does not alter
that boundary. Source-derived candidate / empirical not_run / efficacy
not_evaluated; author reports are not local quality or service results.
+## GPU measurement validity and profiler attribution
+
+[GPU measurement validity and profiler attribution](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/gpu-measurement-validity.md)
+extends `inference-benchmark-design` with clock/cache controls, conditional
+within-run attribution, work-count signatures and timing-trap checks. This is
+evaluation guidance, not a hardware-resource optimization mechanism or added
+numbered-layer coverage; the owner's N/A declaration remains applicable.
+Candidate / empirical not_run / efficacy not_evaluated; task-supplied numerical
+observations are unverified and establish no device or service gain.
+
## Embodied.cpp benchmark methodology
The [inference-benchmark-design](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md)
diff --git a/docs/skill-coverage.md b/docs/skill-coverage.md
index fcc1c1d2f..bcd78b5ae 100644
--- a/docs/skill-coverage.md
+++ b/docs/skill-coverage.md
@@ -4,7 +4,7 @@ Generated from the registry, the skill sidecars and the two maintained
matrices. **Do not edit by hand**; regenerate after any registry, skill,
matrix or frozen-plan change.
-- Input digest: `sha256:8f220788fd2d046f894d60991d7fa77c2b899705a88cd0970218f99fe73e276c`
+- Input digest: `sha256:057218d1284e26df8db68c1177abd0a2ff73edbb175adc8441fe92a66a79e124`
- Regenerate: `python tools/build_skill_coverage.py`
- Verify without writing: `python tools/build_skill_coverage.py --check`
@@ -132,7 +132,7 @@ optimization is fully covered or that any procedure has measured benefit.
| [choose-parallel-training-plan](../src/inferrsi/knowledge/data/skill_library/methods/choose-parallel-training-plan/SKILL.md) | common | Compare stage, device-mesh and operator-sharding candidates for static distributed training while accounting for ... | Comparing stage, device-mesh and operator-sharding candidates for static distributed ... | failure_modes, input_contract, produced_artifact, refusal_point
3 advice | 1 Accelerator · Diagnose / Configure / Optimize / Verify
4 Network · Diagnose / Configure / Optimize / Verify | 1 · Systems sources (Alpa — [2201.12023v3](https://arxiv.org/abs/2201.12023v3)) | candidate · not_run · 1 readset(s) | no-eval |
| [dense-performance-evidence](../src/inferrsi/knowledge/data/skill_library/methods/dense-performance-evidence/SKILL.md) | common | Taxonomy of performance evidence an optimization agent needs, ordered by collection cost and diagnostic power. Maps ... | Deciding which performance evidence to collect next and which claim each evidence level ... | failure_modes, input_contract, produced_artifact, refusal_point
4 advice | 1 Accelerator · Observe / Profile / Verify
2 RAM/Host · Verify
4 Network · Verify | 1 · Systems sources (NVIDIA/TensorRT-LLM @ f33e0e5fd)
2 · Methodology and corpus references (JiusiServe/vllm-evolve @ 43cc82c)
2 · Methodology and corpus references (microsoft/SkillOpt @ 343db22) | candidate · not_run · 0 readset(s) | — |
| [evaluate-elastic-serving-placement](../src/inferrsi/knowledge/data/skill_library/methods/evaluate-elastic-serving-placement/SKILL.md) | common | Compare migrating an LLM request between model replicas with changing its sequence-parallel execution group, including ... | Comparing migration of a running LLM request between compatible model replicas with ... | failure_modes, input_contract, produced_artifact, refusal_point
3 advice | 1 Accelerator · Diagnose / Configure / Optimize / Verify
2 RAM/Host · Diagnose / Configure / Optimize / Verify
4 Network · Diagnose / Configure / Optimize / Verify | 1 · Systems sources (Llumnix — [2406.03243v1](https://arxiv.org/abs/2406.03243v1)) | candidate · not_run · 1 readset(s) | no-eval |
-| [inference-benchmark-design](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md) | common | Select task quality gates, reference parity checks and performance workloads for text, agentic coding, work agents ... | Selecting task quality gates, reference parity checks and performance workloads for text ... | failure_modes, input_contract, produced_artifact, refusal_point
8 advice | N/A: SKILL.md §§ Select the task and its evaluation contract, Statistical requirements, Shared-machine measurement discipline, and Search integrity and final-state verification design comparable evaluation; hardware identity and allocation isolation are comparison controls, not a hardware-resource mechanism. | 2 · Methodology and corpus references (JiusiServe/vllm-evolve @ 43cc82c)
2 · Methodology and corpus references ([Neroued/ninfer](https://github.com/Neroued/ninfer/tree/1882c6adbc9804 ...)
2 · Methodology and corpus references ([Xe-Forge repo + paper v1](https://github.com/IntelLabs/Xe-Forge/tree/ ...)
Task-specific benchmark selection (issue #60) ([Embodied.cpp repo](https://github.com/SEU-PAISys/Embodied.cpp/tree/15 ...)
Task-specific benchmark selection (issue #60) ([Issue #60 report](https://github.com/InferMatrixLab/InferRSI/issues/6 ...)
Task-specific benchmark selection (issue #60) ([NVIDIA/NeMo-Speech.cpp @ 20999cc9](https://github.com/NVIDIA/NeMo-Spe ...)
Task-specific benchmark selection (issue #60) ([yuekaizhang/faster-cosyvoice @ f1049c59](https://github.com/yuekaizha ...) | candidate · not_run · 15 readset(s) | — |
+| [inference-benchmark-design](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md) | common | Select task quality gates, reference parity checks and performance workloads for text, agentic coding, work agents ... | Selecting task quality gates, reference parity checks and performance workloads for text ... | failure_modes, input_contract, produced_artifact, refusal_point
9 advice | N/A: SKILL.md §§ Select the task and its evaluation contract, Statistical requirements, Shared-machine measurement discipline, and Search integrity and final-state verification design comparable evaluation; hardware identity and allocation isolation are comparison controls, not a hardware-resource mechanism. | 2 · Methodology and corpus references (JiusiServe/vllm-evolve @ 43cc82c)
2 · Methodology and corpus references ([Neroued/ninfer](https://github.com/Neroued/ninfer/tree/1882c6adbc9804 ...)
2 · Methodology and corpus references ([Xe-Forge repo + paper v1](https://github.com/IntelLabs/Xe-Forge/tree/ ...)
GPU measurement validity and profiler attribution (NVIDIA NCU 2025.3, Nsight Systems 2025.3, CUDA 12.9.1/10.2 archives ...)
Task-specific benchmark selection (issue #60) ([Embodied.cpp repo](https://github.com/SEU-PAISys/Embodied.cpp/tree/15 ...)
Task-specific benchmark selection (issue #60) ([Issue #60 report](https://github.com/InferMatrixLab/InferRSI/issues/6 ...)
Task-specific benchmark selection (issue #60) ([NVIDIA/NeMo-Speech.cpp @ 20999cc9](https://github.com/NVIDIA/NeMo-Spe ...)
Task-specific benchmark selection (issue #60) ([yuekaizhang/faster-cosyvoice @ f1049c59](https://github.com/yuekaizha ...) | candidate · not_run · 16 readset(s) | — |
| [inference-optimization-casebook](../src/inferrsi/knowledge/data/skill_library/methods/inference-optimization-casebook/SKILL.md) | common | How to structure and use a casebook of past optimization decisions as a decision-support tool. Extracts the casebook ... | Structuring and querying a casebook of past optimization decisions as decision support ... | failure_modes, input_contract, produced_artifact, refusal_point
4 advice | 1 Accelerator · Optimize | 1 · Systems sources (NVIDIA/TensorRT-LLM @ f33e0e5fd) | candidate · not_run · 0 readset(s) | no-eval |
| [kernel-writing-guide](../src/inferrsi/knowledge/data/skill_library/methods/kernel-writing-guide/SKILL.md) | common | Decision guide for writing custom inference kernels: when to write one, which framework to choose (Triton, CuTe DSL ... | Deciding whether to write a custom inference kernel, routing a kernel pattern to Triton ... | failure_modes, input_contract, produced_artifact, refusal_point
6 advice | 1 Accelerator · Optimize / Diagnose / Configure / Verify | 1 · Systems sources (Dao-AILab/causal-conv1d @ `cd81f0413cad2fc1e6f17e785ac39f59aae690cd` ...)
1 · Systems sources (NVIDIA/TensorRT-LLM @ f33e0e5fd)
1 · Systems sources (huggingface/transformers @ `0dad7b822255a0ae261ec45ae937371e859ffd1a` ...)
1 · Systems sources (mindspore-ai/akg @ `5aa15f3fe4d856eb60cd3b27447e04f5f1fb4dff` ...)
1 · Systems sources (sgl-project/sglang @ `16176278d0180f1a1a578e53e6eef432577876c2` ...)
1 · Systems sources (vllm-project/vllm @ `537a79f0fc5ba564d2bef43d54915b2f80f9f77d` (root ...) | candidate · not_run · 7 readset(s) | no-eval |
| [optimization-search-policy](../src/inferrsi/knowledge/data/skill_library/methods/optimization-search-policy/SKILL.md) | common | Policy for how an optimization agent should select, order, and stop optimization experiments. Covers candidate ... | Selecting, ordering and stopping optimization experiments: candidate generation ... | failure_modes, input_contract, produced_artifact, refusal_point
5 advice | 1 Accelerator · Optimize | 2 · Methodology and corpus references (JiusiServe/vllm-evolve @ 43cc82c)
2 · Methodology and corpus references (RRSI — arXiv 2609.24972v3)
2 · Methodology and corpus references (arXiv 2608.01867 (CRISP))
2 · Methodology and corpus references (microsoft/SkillOpt @ 343db22) | candidate · not_run · 9 readset(s) | no-eval |
diff --git a/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json b/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json
index 1b9f6d8e4..01db75a81 100644
--- a/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json
+++ b/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json
@@ -1,7 +1,7 @@
{
"schema_version": 1,
"repository": "https://github.com/InferMatrixLab/InferRSI",
- "source_revision": "0143bc4e0228866a07cdc6c94e11fe0d76b93ab2",
+ "source_revision": "ee8c924f61f0a8f17338b8b25909a410398067ad",
"registry": {
"path": "src/inferrsi/knowledge/data/skill_library/skill-index.json",
"sha256": "7904275c3383380db179287171e1430617aaa9ac4f439291eff8a71426987fbb"
diff --git a/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/common/skills/inference-benchmark-design.json b/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/common/skills/inference-benchmark-design.json
index 83e92bbfb..9c4e06251 100644
--- a/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/common/skills/inference-benchmark-design.json
+++ b/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/common/skills/inference-benchmark-design.json
@@ -38,7 +38,10 @@
"vla",
"embodied",
"action-chunks",
- "closed-loop"
+ "closed-loop",
+ "gpu-timing",
+ "profiler-attribution",
+ "measurement-validity"
],
"origin": [
"external"
@@ -49,8 +52,8 @@
},
"checkout_ref": "checkout:inference-benchmark-design",
"source_files": {
- "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md": "11d8d5bbc52d1a39ac4b2f5e659be5246f10662b673ce6e2bb66d63277501a4d",
- "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/metadata.json": "d2b216ff9e2a6de21a401b7f0c4bbe17483233120e3951721f3ce838816f9aac",
+ "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md": "7782936c8b43dc6d27f416da6aa7162a397e3321adc044e8e7515a14791b720b",
+ "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/metadata.json": "bffa79dd93915986252728939f6cba16ee721164910c58ab1f78a2e69299db1d",
"src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/autotunebench-source.json": "bc343ece2bf40c4e8aa4bd8d84f32465d00633255557feda5fe7799ce24e8688",
"src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/autotunebench-trial-validity.md": "cb3a7a5d94d056f868af07f92a16a6fef31eafd3199ed07ae79e93798e320b1d",
"src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/benchmark-commands.md": "0898424e0bff3a6246e1eeeda6abd6c5bd974c047fe32f3bbd5ab9d31ce17cd3",
@@ -59,6 +62,7 @@
"src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/embodied-cpp-boundary-and-quality.md": "0bc3ca87159cce960c7b32f62b82c275f850b98077ae07da0481bd554a3cd71d",
"src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/embodied-cpp-source.json": "056985be689554570a9f12b8e228ce71bdca5be940f9a2d7f6e1963e15a48130",
"src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/faster-cosyvoice-source.json": "9efe8918208adddfb9a31d2abed080923a5ae0dfb6332438179af0de66f54e13",
+ "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/gpu-measurement-validity.md": "96e273d8a193b62540962084cdca7cb4d93ef43473a38b1dcb7228088de793aa",
"src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/justgoal-evaluation-contracts.md": "d25df795fd301e8a1bb436e5f8122d0bac2441c43104e59a3b310fd5a64f2f3e",
"src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/justgoal-source.json": "60f2784384ea1a3b08a2ecafb60207d8e3707247dc7bc177b95b463d81f1cc16",
"src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/ninfer-source.json": "ccda0fc6781e3c4dceb41521ab952445840b4afcc9f5a7d968857f76e8db6272",
@@ -99,7 +103,10 @@
"vla",
"embodied",
"action-chunks",
- "closed-loop"
+ "closed-loop",
+ "gpu-timing",
+ "profiler-attribution",
+ "measurement-validity"
],
"status": "candidate",
"profile": "local-skill-v1",
@@ -485,14 +492,101 @@
"readset": "references/embodied-cpp-source.json",
"license": "CC-BY-4.0",
"scope": "Independently normalized VLA metric semantics; predates the later SmolVLA experiment"
+ },
+ {
+ "type": "document",
+ "url": "https://docs.nvidia.com/nsight-compute/2025.3/ProfilingGuide/index.html",
+ "version": "Nsight Compute 2025.3",
+ "sha256": "c9f3b812ce50a46a751f2c92eff628177554f4f6678bb3381468b1dac91b607e",
+ "readset": "references/gpu-measurement-validity.md",
+ "scope": "Inspected Replay, Overhead, Hardware Model, Metrics Structure, Range and Precision, Reproducibility, Clock Control and Cache Control; source contracts only, no profiler execution."
+ },
+ {
+ "type": "document",
+ "url": "https://docs.nvidia.com/nsight-compute/2025.3/NsightComputeCli/index.html",
+ "version": "Nsight Compute 2025.3",
+ "sha256": "c5129a8afe2439c04906f19312393972ae123062d2ee0894b3ba2b11af43471e",
+ "readset": "references/gpu-measurement-validity.md",
+ "scope": "Inspected Metric Collection and Profiling option defaults for clock/cache control and metric queries; actual target support must be checked separately."
+ },
+ {
+ "type": "document",
+ "url": "https://docs.nvidia.com/nsight-systems/2025.3/UserGuide/index.html",
+ "version": "Nsight Systems 2025.3",
+ "sha256": "1d42e35005fa9a4cb21b2c65d0018e4b065c3756dbba308dc0c41f79746e751f",
+ "readset": "references/gpu-measurement-validity.md",
+ "scope": "Inspected CLI Profile options, CUDA graph/node tracing and CUDA API/kernel/launch/MemOps statistics reports; absence of NCU replay controls is an author inference, not a launcher guarantee."
+ },
+ {
+ "type": "document",
+ "url": "https://docs.nvidia.com/cuda/archive/12.9.1/cuda-c-best-practices-guide/index.html",
+ "version": "CUDA Toolkit 12.9.1 archive",
+ "sha256": "0fcaf0c361d00de56a9870ba96db871a0b880904121e1dfb215ab9882470120b",
+ "readset": "references/gpu-measurement-validity.md",
+ "scope": "Inspected Assess/Optimize, Application Profiling, Amdahl's Law, Timing and Instruction Throughput; normalization/adoption procedure is an unverified author synthesis."
+ },
+ {
+ "type": "document",
+ "url": "https://docs.nvidia.com/cuda/archive/12.9.1/cuda-c-programming-guide/index.html",
+ "version": "CUDA Toolkit 12.9.1 archive",
+ "sha256": "06499a0c6bfc6925cf54c76f6ba86a4986a081c44c602e10d4ef3b4f33a33865",
+ "readset": "references/gpu-measurement-validity.md",
+ "scope": "Inspected Concurrent Kernel Execution, CUDA Graphs (structure, definition/instantiation/execution) and Events; no constant launch/replay overhead or device test claimed."
+ },
+ {
+ "type": "document",
+ "url": "https://docs.nvidia.com/cuda/archive/10.2/nsight-compute/docs/NsightComputeCli/index.html",
+ "version": "CUDA Toolkit 10.2 archive, Nsight Compute CLI",
+ "readset": "references/gpu-measurement-validity.md",
+ "scope": "Inspected nvprof Metric Comparison for dram__bytes_read.sum, dram__bytes_write.sum and smsp__inst_executed.sum. Legacy names do not guarantee current chip/tool availability; no raw-byte digest retained for this browser read."
+ },
+ {
+ "type": "document",
+ "url": "https://docs.nvidia.com/deploy/nvidia-smi/index.html",
+ "version": "Unversioned NVIDIA System Management Interface page, retrieved 2026-11-15; content pinned by SHA-256",
+ "sha256": "94d1737cdff11b5ef89b121e97365706feeb8ccc18978fecec571bdbbe6ec9ed",
+ "readset": "references/gpu-measurement-validity.md",
+ "scope": "Inspected Performance State, Clocks Event Reasons, clocks, temperature and power reporting. Public page may change; digest identifies inspected HTML, which is not redistributed."
+ },
+ {
+ "type": "git",
+ "url": "https://github.com/triton-lang/triton",
+ "commit": "9641643da6c52000c807b5eeed05edaec4402a67",
+ "paths": [
+ "python/triton/testing.py"
+ ],
+ "readset": "references/gpu-measurement-validity.md",
+ "license": "MIT",
+ "scope": "v3.2.0 do_bench: write-based cache.zero_ flush, preliminary estimate and per-sample event boundaries; tag resolved with git ls-remote, source inspected in public browser. No flush comparison run."
+ },
+ {
+ "type": "document",
+ "url": "https://docs.pytorch.org/docs/2.7/cpp_extension.html",
+ "version": "PyTorch 2.7",
+ "readset": "references/gpu-measurement-validity.md",
+ "scope": "Inspected load/load_inline and the extension name, build-directory and pybind11 module contract. Distinct names for separately loaded variants are author advice, not a documented universal duplicate-name failure; no raw-byte digest retained for this browser read."
+ },
+ {
+ "type": "paper",
+ "url": "https://arxiv.org/html/2509.14279v1",
+ "id": "arxiv:2509.14279",
+ "version": "v1",
+ "sections": [
+ "3",
+ "Appendix A",
+ "Appendix C.3"
+ ],
+ "readset": "references/gpu-measurement-validity.md",
+ "license": "CC-BY-4.0",
+ "scope": "Lange et al., Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimization. Inspected benchmark loopholes, compromised-task filtering/examples and eager/compiled baseline comparisons; reports remain author-reported, not locally reproduced."
}
],
"empirical_validation": "not_run",
"applicability": {
- "scope": "Selecting task quality gates, reference parity checks and performance workloads for text, agentic coding, work agents, cache-aware ASR, TTS, full-duplex audio/video, image and video inference, and designing goodput-based multi-regime comparisons that fit InferRSI's evidence trail; interpreting treatment delivery, arm validity and stated-versus-observed protocol enforcement in supplied tuning reports; interpreting source-bound kernel trial origin, host evaluator outcomes, unresolved performance and measured-only versus manual candidate selection; designing and interpreting the selected NInfer hybrid speculative state-restoration and completion validation contract; designing or auditing VLA deployment comparisons with explicit observation-to-action boundaries, generated/replayed horizons, per-task quality and source/precision identity",
+ "scope": "Selecting task quality gates, reference parity checks and performance workloads for text, agentic coding, work agents, cache-aware ASR, TTS, full-duplex audio/video, image and video inference, and designing goodput-based multi-regime comparisons that fit InferRSI's evidence trail; interpreting treatment delivery, arm validity and stated-versus-observed protocol enforcement in supplied tuning reports; interpreting source-bound kernel trial origin, host evaluator outcomes, unresolved performance and measured-only versus manual candidate selection; designing and interpreting the selected NInfer hybrid speculative state-restoration and completion validation contract; designing or auditing VLA deployment comparisons with explicit observation-to-action boundaries, generated/replayed horizons, per-task quality and source/precision identity; assessing GPU timing validity and profiler-based attribution of kernel/inference gains",
"prerequisites": "The primary objective, timing boundary, quality gates, sample/failure accounting, required regimes, repetitions, seeds and decision rule are frozen before optimization; baseline and candidate share hardware, software stack, input distribution, benchmark configuration, warmup and measurement protocol, and on a shared machine also one time window and dedicated devices; cache-aware ASR also requires the selected model/config and first/steady geometry, decoded clock and reset/EOU/EOF boundaries to be established; VLA comparisons also freeze generated/replayed horizons, initial-state/noise protocol and intended precision or quantization treatment",
"input_contract": "The task family and deployment use case; candidate quality evaluators and reference parity; the declared SLO and the caller's policy on which regimes must improve or must not regress; for an audit, a supplied fixed-workload report with request-to-trial grouping, exclusion and failure counts and the declared aggregation rule; for a treatment audit, the frozen delivery/arm-validity rules, run/candidate identities, feedback and consumer-input evidence, and per-arm raw records; for a kernel trial audit, complete saved candidate bytes and IDs, baseline/spec/reference/variant identities, result-origin fields, raw host outcome and return code, named gate and persisted ratio, actual oracle and the selecting/delivering caller; for the NInfer slice, a complete checkpoint/frontier identity, physical verify-block/state oracle, final adopted extents and correlated execution/termination records; for VLA audits, the declared CPU input/output boundary, source/fixture/checkpoint/processor identities, per-task trial scope and available raw records or explicit artifact gaps",
- "produced_artifact": "A frozen benchmark design with evaluators, reference parity, primary objective, timing boundary, quality gates, required regimes, repetition and seed rules and an explicit decision config, plus the evidence-tree layout a PASS claim must produce; for a supplied report, the computed threshold result and missing evidence. Candidate knowledge, not qualification of any model or backend; for a treatment audit, a contract-bound eligibility and interpretation report separating treatment exposure, observed checks, comparator identity, anomaly disposition and source conflicts; for a kernel trial audit, a source-bound interpretation separating artifact/cohort identity, origin, oracle, performance resolution and selected-versus-delivered eligibility, without promoting kernel parity into a gain or whole-model acceptance; for the NInfer slice, a source-bound validation plan or report interpretation that separates state-oracle evidence, final adoption, retirement and workload denominators; for VLA reports, a timing/denominator and precision reconciliation with per-task quality/completion accounting and source-artifact limits",
+ "produced_artifact": "A frozen benchmark design with evaluators, reference parity, primary objective, timing boundary, quality gates, required regimes, repetition and seed rules and an explicit decision config, plus the evidence-tree layout a PASS claim must produce; for a supplied report, the computed threshold result and missing evidence. Candidate knowledge, not qualification of any model or backend; for a treatment audit, a contract-bound eligibility and interpretation report separating treatment exposure, observed checks, comparator identity, anomaly disposition and source conflicts; for a kernel trial audit, a source-bound interpretation separating artifact/cohort identity, origin, oracle, performance resolution and selected-versus-delivered eligibility, without promoting kernel parity into a gain or whole-model acceptance; for the NInfer slice, a source-bound validation plan or report interpretation that separates state-oracle evidence, final adoption, retirement and workload denominators; for VLA reports, a timing/denominator and precision reconciliation with per-task quality/completion accounting and source-artifact limits; for GPU attribution, a source-grounded measurement plan or interpretation binding environment, paired timing, work counts, predicted signature and E2E decomposition with explicit assumptions and unresolved evidence",
"failure_modes": [
"Raw throughput is optimized in place of goodput, so requests served outside the SLO count toward a win",
"One concurrency level or one undisclosed aggregate score stands in for all required regimes",
@@ -514,7 +608,9 @@
"Matching structural pair fields are taken to certify matched effective compute precision or quality equivalence",
"An aggregate VLA success rate conceals task-specific losses or missing/skipped episodes",
"A public current script, recorded experiment revision or binary hash is treated as the exact measured dirty source patch",
- "Whole-device fallback memory is relabelled as process-attributed memory or compared with simulator-inclusive/allocator quantities"
+ "Whole-device fallback memory is relabelled as process-attributed memory or compared with simulator-inclusive/allocator quantities",
+ "NCU replay durations are substituted for normal E2E evidence, or an untouched region is treated as a universal clock without checking common drift and critical-path assumptions",
+ "Unexplained gains, changed baseline modes or same-named JIT loads are adopted without binding actual work, module identity and predicted profiler signatures"
],
"refusal_point": "Records missing coverage and high-variance results as INCONCLUSIVE rather than a win; a run without a frozen RunSpec, or whose validator changed after the plan was frozen, is INCONCLUSIVE regardless of metric values; sets no universal numeric gate or noise threshold; withholds causal treatment benefit/null conclusions when required delivery or arm eligibility is unverified, and preserves conflicts between stated protocol, exported code and source reports; withholds a kernel gain for unresolved performance or an unbound comparison and identifies missing independent semantic evidence when only baseline agreement is available; withholds native state-correctness and cancellation/reuse conclusions when the required same-block oracle, checkpoint identity or matching receipt is missing; withholds VLA quality-preservation, runtime-only attribution, control-rate or current-build conclusions when their required precision, per-task, timestamp or measured-source evidence is missing"
},
@@ -526,7 +622,8 @@
"Trace treatment to its eligible consumer and apply the frozen arm-validity precedence before attributing benefit, saturation or a null.",
"Bind the complete saved candidate and comparator cohort; keep result origin, actual oracle, named gate/ratio and delivered eligibility separate.",
"For hybrid speculative comparisons, freeze the coupled restore point, same-physical-block oracle, zero-draft/zero-adoption controls and complete-cohort versus steady-phase denominators before interpreting a result.",
- "Separate complete VLA request cost, generated/replayed action denominators and actual control cadence; retain per-task quality and measured-source/precision limits."
+ "Separate complete VLA request cost, generated/replayed action denominators and actual control cadence; retain per-task quality and measured-source/precision limits.",
+ "For GPU comparisons, retain timestamped clocks/temperature/power, same-run region shares and work counts; test the predicted mechanism signature and paired E2E change before adopting a gain."
],
"limitations": [
"Source-derived candidate; no local efficacy validation.",
@@ -539,7 +636,8 @@
"The Xe-Forge reference is source-only and limited to inspected host/manual/Claude trial paths; named-verdict update predicates and caller defaults remain explicit. Native tests, backend/model/quality/performance/service behavior and agent efficacy were not evaluated, and source/package checks do not promote whole-model support.",
"The NInfer reference inspects selected Qwen3.5 GDN/MTP source contracts and unexecuted test definitions; its historical author measurements use a different engine revision. Native state/model/device/quality/performance and efficacy remain unvalidated.",
"The Faster CosyVoice reference is source-only: server route credit and client simulated playback remain distinct, reported costs/quality retain their boundaries, overall source rights are UNKNOWN, and native/model/device/audio quality/service/performance are not_run with efficacy not_evaluated.",
- "The Embodied.cpp reference is source-derived: its public latency arrays allow arithmetic audit, but raw episode recount, exact measured script/dirty patch, effective dispatch and current model/device/service or efficacy qualification remain unresolved."
+ "The Embodied.cpp reference is source-derived: its public latency arrays allow arithmetic audit, but raw episode recount, exact measured script/dirty patch, effective dispatch and current model/device/service or efficacy qualification remain unresolved.",
+ "The GPU measurement-validity reference is source-only. The within-run normalization and explanation-first adoption procedure is an unverified author proposal; four task-supplied numerical observations have no supplied raw evidence. GPU/profiler, model/quality, L1–L4 and efficacy validation were not run; legacy metric mappings require target-specific queries."
],
"provenance_readsets": [
"references/task-benchmark-selection.md",
@@ -556,7 +654,8 @@
"references/dependency-aware-tts-playout.md",
"references/faster-cosyvoice-source.json",
"references/embodied-cpp-boundary-and-quality.md",
- "references/embodied-cpp-source.json"
+ "references/embodied-cpp-source.json",
+ "references/gpu-measurement-validity.md"
],
"references": [
"../inference-optimization-casebook/cases/admission-drain-occupancy.md",
@@ -568,10 +667,10 @@
"searchspace_not_applicable": "SKILL.md §§ Select the task and its evaluation contract, Statistical requirements, Shared-machine measurement discipline, and Search integrity and final-state verification design comparable evaluation; hardware identity and allocation isolation are comparison controls, not a hardware-resource mechanism."
},
"applicability": {
- "scope": "Selecting task quality gates, reference parity checks and performance workloads for text, agentic coding, work agents, cache-aware ASR, TTS, full-duplex audio/video, image and video inference, and designing goodput-based multi-regime comparisons that fit InferRSI's evidence trail; interpreting treatment delivery, arm validity and stated-versus-observed protocol enforcement in supplied tuning reports; interpreting source-bound kernel trial origin, host evaluator outcomes, unresolved performance and measured-only versus manual candidate selection; designing and interpreting the selected NInfer hybrid speculative state-restoration and completion validation contract; designing or auditing VLA deployment comparisons with explicit observation-to-action boundaries, generated/replayed horizons, per-task quality and source/precision identity",
+ "scope": "Selecting task quality gates, reference parity checks and performance workloads for text, agentic coding, work agents, cache-aware ASR, TTS, full-duplex audio/video, image and video inference, and designing goodput-based multi-regime comparisons that fit InferRSI's evidence trail; interpreting treatment delivery, arm validity and stated-versus-observed protocol enforcement in supplied tuning reports; interpreting source-bound kernel trial origin, host evaluator outcomes, unresolved performance and measured-only versus manual candidate selection; designing and interpreting the selected NInfer hybrid speculative state-restoration and completion validation contract; designing or auditing VLA deployment comparisons with explicit observation-to-action boundaries, generated/replayed horizons, per-task quality and source/precision identity; assessing GPU timing validity and profiler-based attribution of kernel/inference gains",
"prerequisites": "The primary objective, timing boundary, quality gates, sample/failure accounting, required regimes, repetitions, seeds and decision rule are frozen before optimization; baseline and candidate share hardware, software stack, input distribution, benchmark configuration, warmup and measurement protocol, and on a shared machine also one time window and dedicated devices; cache-aware ASR also requires the selected model/config and first/steady geometry, decoded clock and reset/EOU/EOF boundaries to be established; VLA comparisons also freeze generated/replayed horizons, initial-state/noise protocol and intended precision or quantization treatment",
"input_contract": "The task family and deployment use case; candidate quality evaluators and reference parity; the declared SLO and the caller's policy on which regimes must improve or must not regress; for an audit, a supplied fixed-workload report with request-to-trial grouping, exclusion and failure counts and the declared aggregation rule; for a treatment audit, the frozen delivery/arm-validity rules, run/candidate identities, feedback and consumer-input evidence, and per-arm raw records; for a kernel trial audit, complete saved candidate bytes and IDs, baseline/spec/reference/variant identities, result-origin fields, raw host outcome and return code, named gate and persisted ratio, actual oracle and the selecting/delivering caller; for the NInfer slice, a complete checkpoint/frontier identity, physical verify-block/state oracle, final adopted extents and correlated execution/termination records; for VLA audits, the declared CPU input/output boundary, source/fixture/checkpoint/processor identities, per-task trial scope and available raw records or explicit artifact gaps",
- "produced_artifact": "A frozen benchmark design with evaluators, reference parity, primary objective, timing boundary, quality gates, required regimes, repetition and seed rules and an explicit decision config, plus the evidence-tree layout a PASS claim must produce; for a supplied report, the computed threshold result and missing evidence. Candidate knowledge, not qualification of any model or backend; for a treatment audit, a contract-bound eligibility and interpretation report separating treatment exposure, observed checks, comparator identity, anomaly disposition and source conflicts; for a kernel trial audit, a source-bound interpretation separating artifact/cohort identity, origin, oracle, performance resolution and selected-versus-delivered eligibility, without promoting kernel parity into a gain or whole-model acceptance; for the NInfer slice, a source-bound validation plan or report interpretation that separates state-oracle evidence, final adoption, retirement and workload denominators; for VLA reports, a timing/denominator and precision reconciliation with per-task quality/completion accounting and source-artifact limits",
+ "produced_artifact": "A frozen benchmark design with evaluators, reference parity, primary objective, timing boundary, quality gates, required regimes, repetition and seed rules and an explicit decision config, plus the evidence-tree layout a PASS claim must produce; for a supplied report, the computed threshold result and missing evidence. Candidate knowledge, not qualification of any model or backend; for a treatment audit, a contract-bound eligibility and interpretation report separating treatment exposure, observed checks, comparator identity, anomaly disposition and source conflicts; for a kernel trial audit, a source-bound interpretation separating artifact/cohort identity, origin, oracle, performance resolution and selected-versus-delivered eligibility, without promoting kernel parity into a gain or whole-model acceptance; for the NInfer slice, a source-bound validation plan or report interpretation that separates state-oracle evidence, final adoption, retirement and workload denominators; for VLA reports, a timing/denominator and precision reconciliation with per-task quality/completion accounting and source-artifact limits; for GPU attribution, a source-grounded measurement plan or interpretation binding environment, paired timing, work counts, predicted signature and E2E decomposition with explicit assumptions and unresolved evidence",
"failure_modes": [
"Raw throughput is optimized in place of goodput, so requests served outside the SLO count toward a win",
"One concurrency level or one undisclosed aggregate score stands in for all required regimes",
@@ -593,7 +692,9 @@
"Matching structural pair fields are taken to certify matched effective compute precision or quality equivalence",
"An aggregate VLA success rate conceals task-specific losses or missing/skipped episodes",
"A public current script, recorded experiment revision or binary hash is treated as the exact measured dirty source patch",
- "Whole-device fallback memory is relabelled as process-attributed memory or compared with simulator-inclusive/allocator quantities"
+ "Whole-device fallback memory is relabelled as process-attributed memory or compared with simulator-inclusive/allocator quantities",
+ "NCU replay durations are substituted for normal E2E evidence, or an untouched region is treated as a universal clock without checking common drift and critical-path assumptions",
+ "Unexplained gains, changed baseline modes or same-named JIT loads are adopted without binding actual work, module identity and predicted profiler signatures"
],
"refusal_point": "Records missing coverage and high-variance results as INCONCLUSIVE rather than a win; a run without a frozen RunSpec, or whose validator changed after the plan was frozen, is INCONCLUSIVE regardless of metric values; sets no universal numeric gate or noise threshold; withholds causal treatment benefit/null conclusions when required delivery or arm eligibility is unverified, and preserves conflicts between stated protocol, exported code and source reports; withholds a kernel gain for unresolved performance or an unbound comparison and identifies missing independent semantic evidence when only baseline agreement is available; withholds native state-correctness and cancellation/reuse conclusions when the required same-block oracle, checkpoint identity or matching receipt is missing; withholds VLA quality-preservation, runtime-only attribution, control-rate or current-build conclusions when their required precision, per-task, timestamp or measured-source evidence is missing"
},
@@ -605,7 +706,8 @@
"Trace treatment to its eligible consumer and apply the frozen arm-validity precedence before attributing benefit, saturation or a null.",
"Bind the complete saved candidate and comparator cohort; keep result origin, actual oracle, named gate/ratio and delivered eligibility separate.",
"For hybrid speculative comparisons, freeze the coupled restore point, same-physical-block oracle, zero-draft/zero-adoption controls and complete-cohort versus steady-phase denominators before interpreting a result.",
- "Separate complete VLA request cost, generated/replayed action denominators and actual control cadence; retain per-task quality and measured-source/precision limits."
+ "Separate complete VLA request cost, generated/replayed action denominators and actual control cadence; retain per-task quality and measured-source/precision limits.",
+ "For GPU comparisons, retain timestamped clocks/temperature/power, same-run region shares and work counts; test the predicted mechanism signature and paired E2E change before adopting a gain."
],
"limitations": [
"This generated abstract points to the complete packaged skill; external evidence and upstream repositories remain source links.",
@@ -622,7 +724,8 @@
"The Xe-Forge reference is source-only and limited to inspected host/manual/Claude trial paths; named-verdict update predicates and caller defaults remain explicit. Native tests, backend/model/quality/performance/service behavior and agent efficacy were not evaluated, and source/package checks do not promote whole-model support.",
"The NInfer reference inspects selected Qwen3.5 GDN/MTP source contracts and unexecuted test definitions; its historical author measurements use a different engine revision. Native state/model/device/quality/performance and efficacy remain unvalidated.",
"The Faster CosyVoice reference is source-only: server route credit and client simulated playback remain distinct, reported costs/quality retain their boundaries, overall source rights are UNKNOWN, and native/model/device/audio quality/service/performance are not_run with efficacy not_evaluated.",
- "The Embodied.cpp reference is source-derived: its public latency arrays allow arithmetic audit, but raw episode recount, exact measured script/dirty patch, effective dispatch and current model/device/service or efficacy qualification remain unresolved."
+ "The Embodied.cpp reference is source-derived: its public latency arrays allow arithmetic audit, but raw episode recount, exact measured script/dirty patch, effective dispatch and current model/device/service or efficacy qualification remain unresolved.",
+ "The GPU measurement-validity reference is source-only. The within-run normalization and explanation-first adoption procedure is an unverified author proposal; four task-supplied numerical observations have no supplied raw evidence. GPU/profiler, model/quality, L1–L4 and efficacy validation were not run; legacy metric mappings require target-specific queries."
],
"verification": {
"locally_verified": false,
diff --git a/src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md b/src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md
index a68954bf6..a331d6800 100644
--- a/src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md
+++ b/src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md
@@ -42,6 +42,12 @@ For treatment-enabled tuning comparisons or claimed protocol integrity, read
Trace feedback to the eligible consumer input and apply the frozen arm-validity
precedence before interpreting benefit, saturation or a null.
+For GPU timing comparisons, read
+[measurement validity and profiler attribution](references/gpu-measurement-validity.md).
+Separate normal E2E timing from profiler replay, test within-run attribution
+assumptions and retain work counts plus the predicted mechanism signature.
+This is candidate guidance; its task-supplied observations remain unverified.
+
For hybrid speculative state-restoration or completion evidence, read the
[NInfer validation contract](references/ninfer-state-and-completion-contracts.md).
Freeze a coupled checkpoint and same-physical-block state oracle; distinguish
diff --git a/src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/metadata.json b/src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/metadata.json
index 4e7f7d9b1..0be7cad20 100644
--- a/src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/metadata.json
+++ b/src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/metadata.json
@@ -28,7 +28,10 @@
"vla",
"embodied",
"action-chunks",
- "closed-loop"
+ "closed-loop",
+ "gpu-timing",
+ "profiler-attribution",
+ "measurement-validity"
],
"status": "candidate",
"profile": "local-skill-v1",
@@ -414,14 +417,101 @@
"readset": "references/embodied-cpp-source.json",
"license": "CC-BY-4.0",
"scope": "Independently normalized VLA metric semantics; predates the later SmolVLA experiment"
+ },
+ {
+ "type": "document",
+ "url": "https://docs.nvidia.com/nsight-compute/2025.3/ProfilingGuide/index.html",
+ "version": "Nsight Compute 2025.3",
+ "sha256": "c9f3b812ce50a46a751f2c92eff628177554f4f6678bb3381468b1dac91b607e",
+ "readset": "references/gpu-measurement-validity.md",
+ "scope": "Inspected Replay, Overhead, Hardware Model, Metrics Structure, Range and Precision, Reproducibility, Clock Control and Cache Control; source contracts only, no profiler execution."
+ },
+ {
+ "type": "document",
+ "url": "https://docs.nvidia.com/nsight-compute/2025.3/NsightComputeCli/index.html",
+ "version": "Nsight Compute 2025.3",
+ "sha256": "c5129a8afe2439c04906f19312393972ae123062d2ee0894b3ba2b11af43471e",
+ "readset": "references/gpu-measurement-validity.md",
+ "scope": "Inspected Metric Collection and Profiling option defaults for clock/cache control and metric queries; actual target support must be checked separately."
+ },
+ {
+ "type": "document",
+ "url": "https://docs.nvidia.com/nsight-systems/2025.3/UserGuide/index.html",
+ "version": "Nsight Systems 2025.3",
+ "sha256": "1d42e35005fa9a4cb21b2c65d0018e4b065c3756dbba308dc0c41f79746e751f",
+ "readset": "references/gpu-measurement-validity.md",
+ "scope": "Inspected CLI Profile options, CUDA graph/node tracing and CUDA API/kernel/launch/MemOps statistics reports; absence of NCU replay controls is an author inference, not a launcher guarantee."
+ },
+ {
+ "type": "document",
+ "url": "https://docs.nvidia.com/cuda/archive/12.9.1/cuda-c-best-practices-guide/index.html",
+ "version": "CUDA Toolkit 12.9.1 archive",
+ "sha256": "0fcaf0c361d00de56a9870ba96db871a0b880904121e1dfb215ab9882470120b",
+ "readset": "references/gpu-measurement-validity.md",
+ "scope": "Inspected Assess/Optimize, Application Profiling, Amdahl's Law, Timing and Instruction Throughput; normalization/adoption procedure is an unverified author synthesis."
+ },
+ {
+ "type": "document",
+ "url": "https://docs.nvidia.com/cuda/archive/12.9.1/cuda-c-programming-guide/index.html",
+ "version": "CUDA Toolkit 12.9.1 archive",
+ "sha256": "06499a0c6bfc6925cf54c76f6ba86a4986a081c44c602e10d4ef3b4f33a33865",
+ "readset": "references/gpu-measurement-validity.md",
+ "scope": "Inspected Concurrent Kernel Execution, CUDA Graphs (structure, definition/instantiation/execution) and Events; no constant launch/replay overhead or device test claimed."
+ },
+ {
+ "type": "document",
+ "url": "https://docs.nvidia.com/cuda/archive/10.2/nsight-compute/docs/NsightComputeCli/index.html",
+ "version": "CUDA Toolkit 10.2 archive, Nsight Compute CLI",
+ "readset": "references/gpu-measurement-validity.md",
+ "scope": "Inspected nvprof Metric Comparison for dram__bytes_read.sum, dram__bytes_write.sum and smsp__inst_executed.sum. Legacy names do not guarantee current chip/tool availability; no raw-byte digest retained for this browser read."
+ },
+ {
+ "type": "document",
+ "url": "https://docs.nvidia.com/deploy/nvidia-smi/index.html",
+ "version": "Unversioned NVIDIA System Management Interface page, retrieved 2026-11-15; content pinned by SHA-256",
+ "sha256": "94d1737cdff11b5ef89b121e97365706feeb8ccc18978fecec571bdbbe6ec9ed",
+ "readset": "references/gpu-measurement-validity.md",
+ "scope": "Inspected Performance State, Clocks Event Reasons, clocks, temperature and power reporting. Public page may change; digest identifies inspected HTML, which is not redistributed."
+ },
+ {
+ "type": "git",
+ "url": "https://github.com/triton-lang/triton",
+ "commit": "9641643da6c52000c807b5eeed05edaec4402a67",
+ "paths": [
+ "python/triton/testing.py"
+ ],
+ "readset": "references/gpu-measurement-validity.md",
+ "license": "MIT",
+ "scope": "v3.2.0 do_bench: write-based cache.zero_ flush, preliminary estimate and per-sample event boundaries; tag resolved with git ls-remote, source inspected in public browser. No flush comparison run."
+ },
+ {
+ "type": "document",
+ "url": "https://docs.pytorch.org/docs/2.7/cpp_extension.html",
+ "version": "PyTorch 2.7",
+ "readset": "references/gpu-measurement-validity.md",
+ "scope": "Inspected load/load_inline and the extension name, build-directory and pybind11 module contract. Distinct names for separately loaded variants are author advice, not a documented universal duplicate-name failure; no raw-byte digest retained for this browser read."
+ },
+ {
+ "type": "paper",
+ "url": "https://arxiv.org/html/2509.14279v1",
+ "id": "arxiv:2509.14279",
+ "version": "v1",
+ "sections": [
+ "3",
+ "Appendix A",
+ "Appendix C.3"
+ ],
+ "readset": "references/gpu-measurement-validity.md",
+ "license": "CC-BY-4.0",
+ "scope": "Lange et al., Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimization. Inspected benchmark loopholes, compromised-task filtering/examples and eager/compiled baseline comparisons; reports remain author-reported, not locally reproduced."
}
],
"empirical_validation": "not_run",
"applicability": {
- "scope": "Selecting task quality gates, reference parity checks and performance workloads for text, agentic coding, work agents, cache-aware ASR, TTS, full-duplex audio/video, image and video inference, and designing goodput-based multi-regime comparisons that fit InferRSI's evidence trail; interpreting treatment delivery, arm validity and stated-versus-observed protocol enforcement in supplied tuning reports; interpreting source-bound kernel trial origin, host evaluator outcomes, unresolved performance and measured-only versus manual candidate selection; designing and interpreting the selected NInfer hybrid speculative state-restoration and completion validation contract; designing or auditing VLA deployment comparisons with explicit observation-to-action boundaries, generated/replayed horizons, per-task quality and source/precision identity",
+ "scope": "Selecting task quality gates, reference parity checks and performance workloads for text, agentic coding, work agents, cache-aware ASR, TTS, full-duplex audio/video, image and video inference, and designing goodput-based multi-regime comparisons that fit InferRSI's evidence trail; interpreting treatment delivery, arm validity and stated-versus-observed protocol enforcement in supplied tuning reports; interpreting source-bound kernel trial origin, host evaluator outcomes, unresolved performance and measured-only versus manual candidate selection; designing and interpreting the selected NInfer hybrid speculative state-restoration and completion validation contract; designing or auditing VLA deployment comparisons with explicit observation-to-action boundaries, generated/replayed horizons, per-task quality and source/precision identity; assessing GPU timing validity and profiler-based attribution of kernel/inference gains",
"prerequisites": "The primary objective, timing boundary, quality gates, sample/failure accounting, required regimes, repetitions, seeds and decision rule are frozen before optimization; baseline and candidate share hardware, software stack, input distribution, benchmark configuration, warmup and measurement protocol, and on a shared machine also one time window and dedicated devices; cache-aware ASR also requires the selected model/config and first/steady geometry, decoded clock and reset/EOU/EOF boundaries to be established; VLA comparisons also freeze generated/replayed horizons, initial-state/noise protocol and intended precision or quantization treatment",
"input_contract": "The task family and deployment use case; candidate quality evaluators and reference parity; the declared SLO and the caller's policy on which regimes must improve or must not regress; for an audit, a supplied fixed-workload report with request-to-trial grouping, exclusion and failure counts and the declared aggregation rule; for a treatment audit, the frozen delivery/arm-validity rules, run/candidate identities, feedback and consumer-input evidence, and per-arm raw records; for a kernel trial audit, complete saved candidate bytes and IDs, baseline/spec/reference/variant identities, result-origin fields, raw host outcome and return code, named gate and persisted ratio, actual oracle and the selecting/delivering caller; for the NInfer slice, a complete checkpoint/frontier identity, physical verify-block/state oracle, final adopted extents and correlated execution/termination records; for VLA audits, the declared CPU input/output boundary, source/fixture/checkpoint/processor identities, per-task trial scope and available raw records or explicit artifact gaps",
- "produced_artifact": "A frozen benchmark design with evaluators, reference parity, primary objective, timing boundary, quality gates, required regimes, repetition and seed rules and an explicit decision config, plus the evidence-tree layout a PASS claim must produce; for a supplied report, the computed threshold result and missing evidence. Candidate knowledge, not qualification of any model or backend; for a treatment audit, a contract-bound eligibility and interpretation report separating treatment exposure, observed checks, comparator identity, anomaly disposition and source conflicts; for a kernel trial audit, a source-bound interpretation separating artifact/cohort identity, origin, oracle, performance resolution and selected-versus-delivered eligibility, without promoting kernel parity into a gain or whole-model acceptance; for the NInfer slice, a source-bound validation plan or report interpretation that separates state-oracle evidence, final adoption, retirement and workload denominators; for VLA reports, a timing/denominator and precision reconciliation with per-task quality/completion accounting and source-artifact limits",
+ "produced_artifact": "A frozen benchmark design with evaluators, reference parity, primary objective, timing boundary, quality gates, required regimes, repetition and seed rules and an explicit decision config, plus the evidence-tree layout a PASS claim must produce; for a supplied report, the computed threshold result and missing evidence. Candidate knowledge, not qualification of any model or backend; for a treatment audit, a contract-bound eligibility and interpretation report separating treatment exposure, observed checks, comparator identity, anomaly disposition and source conflicts; for a kernel trial audit, a source-bound interpretation separating artifact/cohort identity, origin, oracle, performance resolution and selected-versus-delivered eligibility, without promoting kernel parity into a gain or whole-model acceptance; for the NInfer slice, a source-bound validation plan or report interpretation that separates state-oracle evidence, final adoption, retirement and workload denominators; for VLA reports, a timing/denominator and precision reconciliation with per-task quality/completion accounting and source-artifact limits; for GPU attribution, a source-grounded measurement plan or interpretation binding environment, paired timing, work counts, predicted signature and E2E decomposition with explicit assumptions and unresolved evidence",
"failure_modes": [
"Raw throughput is optimized in place of goodput, so requests served outside the SLO count toward a win",
"One concurrency level or one undisclosed aggregate score stands in for all required regimes",
@@ -443,7 +533,9 @@
"Matching structural pair fields are taken to certify matched effective compute precision or quality equivalence",
"An aggregate VLA success rate conceals task-specific losses or missing/skipped episodes",
"A public current script, recorded experiment revision or binary hash is treated as the exact measured dirty source patch",
- "Whole-device fallback memory is relabelled as process-attributed memory or compared with simulator-inclusive/allocator quantities"
+ "Whole-device fallback memory is relabelled as process-attributed memory or compared with simulator-inclusive/allocator quantities",
+ "NCU replay durations are substituted for normal E2E evidence, or an untouched region is treated as a universal clock without checking common drift and critical-path assumptions",
+ "Unexplained gains, changed baseline modes or same-named JIT loads are adopted without binding actual work, module identity and predicted profiler signatures"
],
"refusal_point": "Records missing coverage and high-variance results as INCONCLUSIVE rather than a win; a run without a frozen RunSpec, or whose validator changed after the plan was frozen, is INCONCLUSIVE regardless of metric values; sets no universal numeric gate or noise threshold; withholds causal treatment benefit/null conclusions when required delivery or arm eligibility is unverified, and preserves conflicts between stated protocol, exported code and source reports; withholds a kernel gain for unresolved performance or an unbound comparison and identifies missing independent semantic evidence when only baseline agreement is available; withholds native state-correctness and cancellation/reuse conclusions when the required same-block oracle, checkpoint identity or matching receipt is missing; withholds VLA quality-preservation, runtime-only attribution, control-rate or current-build conclusions when their required precision, per-task, timestamp or measured-source evidence is missing"
},
@@ -455,7 +547,8 @@
"Trace treatment to its eligible consumer and apply the frozen arm-validity precedence before attributing benefit, saturation or a null.",
"Bind the complete saved candidate and comparator cohort; keep result origin, actual oracle, named gate/ratio and delivered eligibility separate.",
"For hybrid speculative comparisons, freeze the coupled restore point, same-physical-block oracle, zero-draft/zero-adoption controls and complete-cohort versus steady-phase denominators before interpreting a result.",
- "Separate complete VLA request cost, generated/replayed action denominators and actual control cadence; retain per-task quality and measured-source/precision limits."
+ "Separate complete VLA request cost, generated/replayed action denominators and actual control cadence; retain per-task quality and measured-source/precision limits.",
+ "For GPU comparisons, retain timestamped clocks/temperature/power, same-run region shares and work counts; test the predicted mechanism signature and paired E2E change before adopting a gain."
],
"limitations": [
"Source-derived candidate; no local efficacy validation.",
@@ -468,7 +561,8 @@
"The Xe-Forge reference is source-only and limited to inspected host/manual/Claude trial paths; named-verdict update predicates and caller defaults remain explicit. Native tests, backend/model/quality/performance/service behavior and agent efficacy were not evaluated, and source/package checks do not promote whole-model support.",
"The NInfer reference inspects selected Qwen3.5 GDN/MTP source contracts and unexecuted test definitions; its historical author measurements use a different engine revision. Native state/model/device/quality/performance and efficacy remain unvalidated.",
"The Faster CosyVoice reference is source-only: server route credit and client simulated playback remain distinct, reported costs/quality retain their boundaries, overall source rights are UNKNOWN, and native/model/device/audio quality/service/performance are not_run with efficacy not_evaluated.",
- "The Embodied.cpp reference is source-derived: its public latency arrays allow arithmetic audit, but raw episode recount, exact measured script/dirty patch, effective dispatch and current model/device/service or efficacy qualification remain unresolved."
+ "The Embodied.cpp reference is source-derived: its public latency arrays allow arithmetic audit, but raw episode recount, exact measured script/dirty patch, effective dispatch and current model/device/service or efficacy qualification remain unresolved.",
+ "The GPU measurement-validity reference is source-only. The within-run normalization and explanation-first adoption procedure is an unverified author proposal; four task-supplied numerical observations have no supplied raw evidence. GPU/profiler, model/quality, L1–L4 and efficacy validation were not run; legacy metric mappings require target-specific queries."
],
"provenance_readsets": [
"references/task-benchmark-selection.md",
@@ -485,7 +579,8 @@
"references/dependency-aware-tts-playout.md",
"references/faster-cosyvoice-source.json",
"references/embodied-cpp-boundary-and-quality.md",
- "references/embodied-cpp-source.json"
+ "references/embodied-cpp-source.json",
+ "references/gpu-measurement-validity.md"
],
"references": [
"../inference-optimization-casebook/cases/admission-drain-occupancy.md",
diff --git a/src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/gpu-measurement-validity.md b/src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/gpu-measurement-validity.md
new file mode 100644
index 000000000..8c681316c
--- /dev/null
+++ b/src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/gpu-measurement-validity.md
@@ -0,0 +1,175 @@
+# GPU measurement validity and attribution of gains
+
+Status: source-derived candidate; empirical validation not_run; efficacy
+not_evaluated. Public source observations are cited below. The attribution and
+adoption procedure is an author proposal derived from those sources, unverified
+as a complete method. The final section contains task-supplied author reports
+with unavailable evidence, not locally reproduced cases or efficacy evidence.
+Source versions, inspected sections and available document hashes are retained
+in the owning `metadata.json`; the unversioned SMI page is hash-pinned there.
+
+## Freeze the measurement and environment
+
+1. Bind workload, quality oracle, baseline execution mode, warmup, timed boundary,
+ repetition/aggregation rule, cache policy, streams and graph policy before
+ comparison. Retain paired baseline/candidate samples, including failures;
+ service claims still need the campaign's E2E evidence. This is proposed
+ measurement guidance, not a new numeric gate; see the owning skill's contract.
+2. Record SM/memory clocks, temperature, power draw/limit and throttle reasons
+ alongside timestamped samples; record co-tenancy and allocation changes.
+ Boost state and thermal/power limits can move GPU region times without a
+ code change; competing work can alter scheduling and memory traffic.
+ They need not affect every region equally. E2E time alone cannot attribute
+ a gain to the patch. Sources: [NCG] §§ Clock Control, Reproducibility;
+ [SMI] §§ Performance State, Clocks Event Reasons. Recording is author advice.
+3. Record profiler version, replay mode and actual control settings. Nsight
+ Compute 2025.3 defaults to `--clock-control base` and `--cache-control all`:
+ it attempts base clocks and flushes GPU caches before kernel replay passes.
+ Thermal throttling can still override clock control; platform support varies.
+ Sources: [NCC] § Profiling; [NCG] §§ Clock Control, Cache Control.
+4. Ordinary execution and Nsight Systems tracing do not automatically apply
+ those NCU replay controls. This is an inference from [NCS] § CLI Profile
+ options versus [NCC], not a guarantee about an external launcher. Record
+ external clock locks and explicit cache flushes. Use NCU for isolated kernel
+ diagnosis; its kernel durations are not E2E evidence. Retain separate normal
+ paired E2E measurements and assess profiler overhead. [NCG] §§ Replay,
+ Overhead; [NCS] § CUDA Graph Trace warns of node-tracing overhead.
+
+## Attribute changes within each paired run
+
+Author-proposed diagnostic, derived from [CBP] §§ Application Profiling,
+Amdahl's Law and [NCG] § Clock Control; its reliability is unverified here.
+
+1. Instrument stable regions and retain their times and E2E time from the same
+ run. Use region shares, not a kernel time divided by a different run's total.
+ Keep baseline/candidate interleaved in one allocation. Select an untouched
+ reference region `U` with identical work, placement and dependency behavior.
+2. Compare `region_time / U_time` within each run. `U` acts as an internal clock
+ only if it shares the relevant clock sensitivity and is unaffected by the
+ patch. Verify its counts and environment; use multiple controls if possible.
+ A CPU-bound or differently memory-bound region is not a universal GPU clock.
+ Common movement is a confounding signal, not proof that normalization works.
+3. Predict the E2E change from measured changed-region speedups and baseline
+ shares using an Amdahl decomposition. For disjoint serial regions, with no
+ drift and an unchanged residual, define `f_i = B_i / T_B` and `s_i = B_i/C_i`:
+
+ ```text
+ predicted candidate/baseline time = 1 - sum(f_i) + sum(f_i / s_i)
+ observed candidate/baseline time = T_C / T_B
+ ```
+
+ If explicitly assuming common multiplicative drift `d = U_C/U_B`, use
+ `s_i = (B_i/U_B)/(C_i/U_C)` and compare the prediction to `(T_C/T_B)/d`.
+ That correction assumes the residual shares the drift too; model unaffected
+ host time separately. Retain raw and normalized ratios and their uncertainty.
+4. Check prediction against the observed paired E2E ratio under the frozen
+ campaign uncertainty rule. A mismatch leaves attribution unresolved: inspect
+ noise, overlap/critical-path effects and side effects in other regions.
+ Do not sum overlapping kernel durations as wall time. Use the dependency
+ timeline for concurrent work; a faster off-path kernel need not shorten E2E.
+ Sources for these limits: [CPG] §§ Concurrent Kernel Execution, CUDA Graphs;
+ the mismatch interpretation is an author inference, not a causal proof.
+
+## Retain work counts as well as time
+
+For identical work and comparable collection conditions, these author-selected
+counts are less directly sensitive to clocks/temperature than durations or rates.
+They explain small changes; they do not independently establish a speedup.
+
+| Work signature | Collection and interpretation |
+| --- | --- |
+| Kernel set and launch count | [NCS] CUDA GPU Kernel Summary / launch trace: retain kernel identity and instances per region, not just aggregate time. |
+| DRAM read/write bytes | `dram__bytes_read.sum`, `dram__bytes_write.sum`; [NCM] § nvprof Metric Comparison names these counters. Retain both directions under the same cache/replay policy. |
+| Executed instructions / FLOPs | `smsp__inst_executed.sum` counts warp instructions, not FLOPs ([NCM]). Derive FLOPs only with documented operation, datatype, active-thread and tensor-instruction semantics; count FMA as two operations ([CBP] § Instruction Throughput). |
+| Host↔device copy bytes | [NCS] CUDA GPU MemOps Summary (by Size): separate HtoD/DtoH bytes, instances and transfer types. |
+| Synchronization count | [NCS] CUDA API trace: count named host synchronization calls per region. API calls do not enumerate every device barrier or dependency. |
+| CUDA graph node count | [CPG] § CUDA Graphs: retain graph topology/node types and replay count; [NCS] node tracing exposes activities. A graph launch count is not its node count. |
+
+The metric names above have an official legacy mapping in the CUDA 10.2 NCU
+archive [NCM], not a promise of availability on every current GPU. Query the
+installed NCU/device with `--query-metrics`; resolve suffixes with
+`--query-metrics-mode suffix --metrics ` ([NCC] § Metric Collection).
+Retain supported names, units, collection passes and per-region sums; mark absent
+counters unavailable. Do not replace a count with percent-of-peak or per-second.
+
+Author caveats: these are conditional invariants, not constants. Cache residency,
+co-tenancy and replay can change DRAM traffic; data-dependent branches, spin
+loops and algorithms can change instructions. [NCG] §§ Range and Precision,
+Cache Control describes asynchronous traffic and multi-pass variability.
+For FLOPs, keep the derivation with the raw counters; instruction count alone
+does not distinguish useful arithmetic, address work or redundant computation.
+
+## Require an explanation before adoption
+
+Author-proposed candidate guidance, motivated by [CBP]'s Assess / Optimize
+cycle and [RB] § 3 and Appendix A; no efficacy claim or repository-wide gate.
+
+Before measurement, state a mechanism and predicted signature: region, metric,
+direction and rough size. For example, removing a materialized intermediate
+predicts fewer DRAM bytes in the affected region; fusion predicts fewer launches
+and perhaps less intermediate traffic. These are hypotheses to verify, not
+automatic consequences or numeric thresholds. Retain counters, timelines,
+paired E2E samples, quality results and the Amdahl check together. Adopt only
+when retained data supports the explanation and frozen campaign gates hold.
+Leave unexplained numeric gains unadopted / INCONCLUSIVE for attribution;
+reject a contradicted mechanism/candidate rather than narrating it as a win.
+Distinguish this adoption decision from the campaign's PASS/FAIL/INCONCLUSIVE.
+
+## Audit common timing traps
+
+- **L2 flushing:** Triton v3.2.0 `do_bench` writes a cache buffer with
+ `cache.zero_()` before each measured start event ([TR]). Its preliminary
+ runtime estimate includes the flush, whereas per-sample events surround `fn`.
+ Record the method, buffer size, stream/order and cold-versus-warm intent.
+ Author advice: compare flush methods as a separate control; an excluded flush
+ may still change the next kernel's cache/memory state. Do not infer that a
+ read-based flush is equivalent or superior without device-specific evidence.
+- **Events versus host timers:** CUDA launches are asynchronous. Use synchronized
+ host boundaries for completed wall time; CUDA events timestamp stream progress
+ and require completion before reading elapsed time ([CBP] § Timing).
+ Capture/instantiation, host submission, graph replay and kernel execution have
+ different costs ([CPG] § CUDA Graphs). Author advice: expose fixed launch/replay
+ costs with matched repetition counts; amortize only what deployment amortizes.
+ Events are not pure instruction time when their interval contains GPU idle gaps.
+- **Baseline identity:** state eager versus `torch.compile`, compiler settings,
+ warmup and graph mode. Speedup is `baseline_time/candidate_time`; changing the
+ denominator changes the claim (arithmetic / author advice). [RB] Appendix C.3
+ explicitly distinguishes eager and compiled comparisons; neither is universal.
+- **JIT extension identity:** give separately loaded variants distinct extension
+ names and retain the actual module/function and built-library identity. This
+ is author advice inferred from [PT] `load`: the build directory uses `name`,
+ which must match the pybind11 module. The docs do not establish that every
+ repeated name fails; do not assume two same-named loads are independent.
+- **Benchmark exploits:** [RB] § 3 and Appendix A report inefficient baselines,
+ low-variation outputs and input-specific shortcuts passing exposed checks.
+ Author advice: inspect actual timed work and use independent varied-input
+ quality tests. A correctness flag or extreme ratio alone cannot clear a gain.
+
+## Author-reported observations: unverified
+
+Task-supplied reports only; raw samples, configurations and profiler artifacts
+were not supplied. The numbers below are not public-source findings, reproduced
+results, reusable thresholds or evidence that this procedure is effective.
+
+| Reported scope | Reported observation | Limit |
+| --- | --- | --- |
+| One H200 campaign | Write-based L2 flush inflated timings about 9% versus read-based flushing. | Cause and flush equivalence unverified. |
+| Shared nodes | Same-run interleaved ratios were reported trustworthy while cross-run baselines drifted ±7–15%. | Reliability, uncertainty and common-drift assumptions unverified. |
+| Same code, different weight placement | Timing differed about 4%. | Placement and causal mechanism unverified. |
+| Same candidate, different baseline | 7.35x versus eager, 1.43x versus `torch.compile`. | Baseline configurations, quality and timings unverified. |
+
+Original synthesis; no upstream code, figures or raw reports redistributed.
+NVIDIA/PyTorch document rights remain with their owners; Triton is MIT-licensed;
+Lange et al.'s [RB] is CC BY 4.0. Source review is not GPU execution, model quality,
+L1–L4 qualification, independent review or proof of agent efficacy.
+
+[NCG]: https://docs.nvidia.com/nsight-compute/2025.3/ProfilingGuide/index.html
+[NCC]: https://docs.nvidia.com/nsight-compute/2025.3/NsightComputeCli/index.html
+[NCS]: https://docs.nvidia.com/nsight-systems/2025.3/UserGuide/index.html
+[SMI]: https://docs.nvidia.com/deploy/nvidia-smi/index.html
+[CBP]: https://docs.nvidia.com/cuda/archive/12.9.1/cuda-c-best-practices-guide/index.html
+[CPG]: https://docs.nvidia.com/cuda/archive/12.9.1/cuda-c-programming-guide/index.html
+[NCM]: https://docs.nvidia.com/cuda/archive/10.2/nsight-compute/docs/NsightComputeCli/index.html
+[TR]: https://github.com/triton-lang/triton/blob/9641643da6c52000c807b5eeed05edaec4402a67/python/triton/testing.py
+[PT]: https://docs.pytorch.org/docs/2.7/cpp_extension.html
+[RB]: https://arxiv.org/html/2509.14279v1
diff --git a/tests/check_installed.py b/tests/check_installed.py
index 93102e98b..9eaa188b9 100644
--- a/tests/check_installed.py
+++ b/tests/check_installed.py
@@ -540,7 +540,7 @@ def _inventory(root, prefix):
**_inventory(files("inferrsi").joinpath("knowledge", "data"), "knowledge/data"),
**_inventory(files("inferrsi").joinpath("harness", "data"), "harness/data"),
}
-assert len(inventory) == 927, f"installed resource count is {len(inventory)}, expected 927"
+assert len(inventory) == 929, f"installed resource count is {len(inventory)}, expected 929"
assert sum(name.startswith("harness/data/") for name in inventory) == len(REPO_TYPES)
# Every pinned record path and every registered skill body resolves inside that
# same enumeration, so the inventory and the records describe one resource set.