Skip to content

Extend VLA observation-to-action benchmark guidance - #342

Merged
Dong1017 merged 9 commits into
InferMatrixLab:mainfrom
Dong1017:codex/issue91-embodied-source
Nov 16, 2026
Merged

Dong1017 merged 9 commits into
InferMatrixLab:mainfrom
Dong1017:codex/issue91-embodied-source

Conversation

@Dong1017

@Dong1017 Dong1017 commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

VLA benchmark reports can mistake latency amortized over a generated action chunk for real control cadence, and aggregate success can hide task-specific quality losses. This change adds source-bound observation-to-action, replay-denominator, precision and per-task quality guidance to the existing inference-benchmark-design owner.

Sources are Embodied.cpp at 156bed15, licensed Apache-2.0, and the companion paper v3, CC BY 4.0. The source record identifies 17 inspected files and their exact versions. The later SmolVLA experiment is not presented as a paper-v3 result.

The reference preserves the different measured-script hashes, incomplete dirty patch and absent raw episode records, as well as the reported precision and whole-device memory boundaries. Public sample arithmetic is identified separately from author-reported quality results. The registered owner remains candidate / not_run, its hardware field remains absent, and its original SearchSpace resource exclusion remains intact.

Current-main integration (head 9081da21617f7222559589e6a84296fb5a29ab8f)

The previously approved head f68c9f94 conflicted with main after the K01 (#418) and K02 (#426) moves into src/inferrsi/knowledge. This head merges main in two rounds and fast-forwards from f68c9f94; no history was rewritten. The knowledge content is unchanged; only file locations and link targets follow main.

Commit Parents Content
4b3ad9403c9f863421d1da2b2f2df13492582690 f68c9f94, main aaa17a5d Combined source after the package move, described below. Generated projections kept main's bytes until regeneration
87c84d608a6e8d18c763d5f5d3e7495416d0e340 4b3ad940 Projections regenerated from 4b3ad940
23d27be8bde499f8dadd1436657017cf2f4077ab 87c84d60, main f5362887 Main after #351. Only _provenance.json and docs/skill-coverage.md overlapped; they kept main's bytes until regeneration. This PR's hand-written files are unchanged
352eb0b3639b92a7cf1935886acac3463aec7450 23d27be8 Projections regenerated by main's unchanged writers from 23d27be8
9081da21617f7222559589e6a84296fb5a29ab8f 352eb0b3 tests/check_installed.py counts the two new reference files: the installed resource count rises from 925 to 927, as #351 raised it for its own two references
  • Owner files: main moved inference-benchmark-design to src/inferrsi/knowledge/data/skill_library/ without changing its SKILL.md or metadata.json. This PR's four owner files (SKILL.md, metadata.json and the two new references) are byte-identical to f68c9f94 at the moved paths.
  • docs/reference-matrix.md: the only hand-written conflict. Main's three preceding rows keep main's new link paths, and the Embodied row (1 line) follows them.
  • Links: this PR's 6 skill-library links (3 in the Embodied reference-matrix row, 3 in the Embodied SearchSpace section) now point to src/inferrsi/knowledge/data/skill_library/, and each target exists at this head. Main already had no links to the old path.
  • docs/searchspace-matrix.md: the 13-line Embodied section still follows main's sections at the end of the file.

Each regeneration changed 3 generated files: the inference-benchmark-design record, _provenance.json (now pinned to 23d27be8) and docs/skill-coverage.md. Relative to main f5362887, the head changes 10 paths: the 6 canonical paths, these 3 and tests/check_installed.py.

Commands run at 352eb0b3, all exit 0 (the last commit changes only tests/check_installed.py):

python tools/build_skill_index.py --source-revision 23d27be8bde499f8dadd1436657017cf2f4077ab
python tools/build_skill_coverage.py
python tools/build_skill_index.py --source-revision 23d27be8bde499f8dadd1436657017cf2f4077ab --check   # 80 metadata records current
python tools/build_skill_coverage.py --check   # matches sources

git merge-tree --write-tree f5362887 9081da21 is conflict-free and produces exactly this head's tree 891bf47c672903c1bb48af911b5b615bfa8cfa34.

Tests:

  • Hosted checks at 352eb0b3: full-check (3.10 and 3.11), pr-references, dashboard-browser, plan, publish, declarations and knowledge-model-policy passed. Both installed-package jobs failed only on installed resource count is 927, expected 925, which 9081da21 addresses.
  • Focused run of 14 knowledge, catalog, snapshot and policy test files on Linux (Python 3.12.13, pytest 9.0.3, git 2.34.1) at 352eb0b3: 374 passed and 11 failed. All 11 failures are fixture commands that this git rejects as git --no-replace-objects --no-lazy-fetch ... with usage exit 129.
  • The same files on Windows at 87c84d60: 370 passed, 7 skipped and 8 failed. All 8 failures are fixture subprocesses (git init or a child python) that exited with status 0xC0000142 before any assertion ran, on a host that was short of memory.
  • Every focused test passed on at least one of the two hosts (at their respective heads), except 5 fixture tests in tests/test_source_snapshot.py and 2 in tests/test_knowledge_declaration_paths.py. Neither host could start those, and the hosted full-check at 352eb0b3 covers them.

Previous integration (head f68c9f948875693bddb1da84699cbbcb7849beec)

The following applies to the previous head over main bd61d20c52b4d772a3ea282ae9cb3c0fd50075df and is not substituted for current-head checks.

That head followed the agreed order after #332, #322, #360, #324, #325, #359 and #339 landed, and addressed the cross-PR integration blockers in the earlier review. Source commit 4c778ff95e77dfdcdd869f2e8a17d18d77cc7122 merged main; f68c9f94 regenerated the projections.

Xe-Forge, NInfer and Faster CosyVoice already extend inference-benchmark-design, so four hand-written files needed a real merge. Main's additions come first, then this PR's:

  • metadata.json: merged by JSON field. Main's 7 tags, 4 sources, 5 failure modes, 2 recommendations, 3 limitations, 6 readsets and the three applicability clauses stay; this PR appends its 4 tags, 2 sources, 5 failure modes, 1 recommendation, 1 limitation, 2 readsets and its clause on each of scope, input contract, produced artifact and refusal point. The merge stops on any concurrent edit that is not a pure append.
  • SKILL.md: the 7-line observation-to-action paragraph is added after the Faster CosyVoice paragraph. This PR also replaces the one-line description; main had not changed that line, so the replacement applies unchanged.
  • docs/reference-matrix.md: the Embodied row (1 line) follows main's rows.
  • docs/searchspace-matrix.md: the 13-line Embodied section follows main's sections at the end of the file.

The two new reference files were byte-identical to the head before. With the shared source snapshot from #359, regeneration changed 3 generated files, and the head changed 9 paths relative to that main. Native index and coverage write/check modes passed from 4c778ff9, and git merge-tree --write-tree bd61d20c f68c9f94 produced exactly that head's tree 0c01f6c263c2d1c59c677eb207136efcb936d0e0. A local focused test run was not repeated at that head because the host stopped it for low memory; hosted Python 3.10/3.11 full and installed-package checks were the test evidence. amy-why-3459 and cursor[bot] approved f68c9f94.

Earlier head evidence (0f4433b7c9bbc1512a4ef5526f087c2a9730d239)

The following results apply to the previous head over main 663f3b32 and are not substituted for current-head checks.

Canonical source M: d5fc2121d6d1628111403ba70ec3b70638ab8674. Native generated G and head: 0f4433b7c9bbc1512a4ef5526f087c2a9730d239.

Native index and coverage write/check modes passed from the actual M. At that head, complete checkout and source-package resource reads matched the committed bytes; the generated index and coverage were verified against the actual sources. An independent Sol/ultra review of that complete head found no blocking source or generated-projection issue. Ready CI passed against a tested merge whose tree equals that head's tree. Python 3.10 and 3.11 each reported 3,544 passed, 13 skipped, 1 warning and 284 subtests; both isolated installed-wheel checks read 127 source-grounded records.

Limits

The source-derived candidate does not establish upstream runtime, model, robot, device, quality, performance or agent-efficacy results. The snapshot pin 23d27be8 is a branch commit; after a squash merge it is not an ancestor of main, and the next knowledge PR re-pins when it regenerates.

Refs #91. Refs #177.

Knowledge authorship declaration

{
  "schema_version": 1,
  "head_sha": "9081da21617f7222559589e6a84296fb5a29ab8f",
  "human_only": false,
  "material_ai_authors": [
    {
      "tool": "Codex",
      "model_id": "gpt-6.1-sol",
      "effort": "max",
      "role": "author"
    },
    {
      "tool": "Claude Code",
      "model_id": "claude-opus-5-5",
      "effort": "xhigh",
      "role": "author"
    }
  ]
}

Describe observation-to-action timing, replay denominators, precision
and per-task quality evidence using the existing benchmark owner.

Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com>
Agent-assisted-by: Codex / gpt-6.1-sol [effort=max] (author)
Build the native catalog and coverage from source revision
d5fc212.

Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com>
Agent-assisted-by: Codex / gpt-6.1-sol [effort=max] (author)
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-04T04:57:27.059898Z 0f4433b PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Skill evaluation

PLANNED: not executed; there are no task outcomes yet.
No approval, model efficacy or current-head test result is implied.

  • Next: run the pinned source/index --check below at head 9081da21617f7222559589e6a84296fb5a29ab8f.
  • Blockers: Executable snapshot entries are not supported. No unsafe fallback; details below.
  • Selected comparisons: 0 · Estimated task calls: 0. Uncovered/unsupported changes: 10; Canonical library paths: 4; Generated projection paths: 2; Other paths: 4. These are paths, not independent claims.
  • Return: command, input/source pin, exit status, outputs/failures and untested claims for independent verified/refuted/unverified disposition.
Author verification steps (nonpaid; not yet run)
  1. Bind the checkout. Run git rev-parse HEAD; it must equal 9081da21617f7222559589e6a84296fb5a29ab8f. If it differs, obtain a current plan; do not reuse old results.
  2. Check source/index consistency. At that PR checkout, using the reviewed repository tool and its documented development dependencies, run:
python tools/build_skill_index.py --source-revision 23d27be8bde499f8dadd1436657017cf2f4077ab --check

The declared source pin matches this head's canonical library bytes. The command has not run: retain its exit code and full drift/success output. It checks the generated projection, not claim truth or skill efficacy.
No supported automatic comparison is selected. Review the canonical changes and explain which claims need evidence, are unaffected, or remain unverified; do not treat generated records as separate behavioral claims.

  • inference-benchmark-design
    Selected by the trusted path mapping: src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md; src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/metadata.json (+2 more in the plan).
    Unsupported: Executable snapshot entries are not supported. No snapshot prepared; task metadata is in details. Do not relax snapshot safety.
    Uncovered scope by location: Canonical library paths: 4; Generated projection paths: 2; Other paths: 4. Path counts are not claim counts; no metadata-only conclusion is inferred.
  1. Return evidence for independent review: head, check/task/source pin, command, environment, exit status, outputs/failures and untested claims. Label scripted controls. Request each claim's disposition: verified, refuted or unverified; this plan proves neither model provenance nor benefit.
    Check recipes and evidence limits

Evidence and logs · Retention: 30 days.
Retained artifacts are verifiable only before Unix time 1797399502 while the linked artifact is available. Expired evidence is unverified.

Plan bindings, budget, frozen exam and advisory details

Advisory: obtain missing evidence

Retained evidence availability: planned / unexecuted. No approval or merge authorization is issued.
Applicable only while these revisions and inputs match the current PR.
Paid execution requires trusted label authorization and cumulative PR reservation. Unknown accounting blocks new paid calls; planning is not execution.
Pipeline evidence only; offline fixed outputs do not establish skill efficacy.
Unchanged-correct is valid neutral evidence, not incremental benefit.
Content/source review and authorship admission remain separate.
Trusted regrading checks retained answers, not model/hardware provenance or a hostile executor; this is not an efficacy certification.

  • Retained evidence is planned / unexecuted.
  • No supported comparison cases were selected.
  • Uncovered/unsupported scope needs a case or explicit reviewer disposition.
    Plan SHA-256: d2458feecdeb5d310ca68e54fd61b8d2a8158a45bbde4bc60c89c95f7039b8a9
    Base: f536288710c726a827bfa3e4357c2c911984cbc3 · Head: 9081da21617f7222559589e6a84296fb5a29ab8f
    Trusted implementation: f536288710c726a827bfa3e4357c2c911984cbc3

Model/settings: {"effort":"not_applicable","name":"claude-opus-5-5"}
Budget: {"comparisons":0,"estimated_calls":0,"limits_per_comparison":{"campaign_seconds":600,"max_task_calls":2,"repeats":1,"timeout_seconds":180,"token_stop_new_trials":200000},"provider_config_sha256":"b28d80f8194fbe0f5a9f01515efb24b218979fd45f702776560da276f4b30d04","reservation":null}

Skill Comparison Cases Estimated calls
inference-benchmark-design unsupported snapshot: Executable snapshot entries are not supported — 0

Selection/task metadata (not a relevance verdict):

{"adapter":"audit","basis":"trusted_path_mapping","matched_paths":["src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md","src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/metadata.json","src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/embodied-cpp-boundary-and-quality.md","src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/embodied-cpp-source.json"],"tasks":[{"evidence_scope":"http_fixed_workload","id":"audit-threshold","request":"Audit the fixed input report using its contract. Compute the median of per-trial request means, including every request in each trial, and the percentage reduction relative to the baseline aggregate.","split":"holdout"}]}

Automatic execution / cumulative PR budget:

{"reason":"Trusted configuration or pricing missing, invalid or expired","remaining_usd":"unknown","reserved_or_unresolved_usd":"unknown","spent_or_legacy_usd":"unknown","state":"blocked"}

Projection check selection (not an executed result):

{"declared_in":"src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json","source_revision":"23d27be8bde499f8dadd1436657017cf2f4077ab","status":"ready"}

Provider/settings: {"client":{"api_version":"2023-06-01","endpoint":"https://yibuapi.com/v1/messages","kind":"anthropic-messages-full-context-v1","max_output_tokens":4096,"max_read_bytes":32768,"max_request_bytes":262144,"max_requests":1,"max_tool_calls":12,"provider":"yibuapi"},"limits":{"token_stop_new_trials":200000},"model":{"effort":"not_applicable","name":"claude-opus-5-5"},"pricing":{"currency":"USD","expires_at":1793271228,"group":"default","max_input_tokens":1000000,"model":"claude-opus-5-5","rate_evidence":"https://github.com/InferMatrixLab/InferRSI/issues/215#issuecomment-6467531341","rates":{"cache_read":"0.4","cache_write":"10","input":"8","output":"40"},"request_fee":"0","unit":"per_million_tokens"},"spend_limit":{"amount":"200","currency":"USD"}}

Uncovered / unsupported paths (10)
  • uncovered: "docs/reference-matrix.md"
  • uncovered: "docs/searchspace-matrix.md"
  • uncovered: "docs/skill-coverage.md"
  • uncovered: "src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json"
  • uncovered: "src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/common/skills/inference-benchmark-design.json"
  • unsupported: "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md"
  • unsupported: "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/metadata.json"
  • unsupported: "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/embodied-cpp-boundary-and-quality.md"
  • unsupported: "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/embodied-cpp-source.json"
  • uncovered: "tests/check_installed.py"

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Source-only LGTM for the bounded #91 VLA benchmark-guidance amendment at 0f4433b7c9bbc1512a4ef5526f087c2a9730d239, against main 663f3b327998526a13aa31e72ad729025c18983c. No blocking change found in this scope.

Existing review disposition: refreshed the conversation, submitted reviews and inline threads before publication. There are no existing submitted reviews or inline threads to resolve; the automated current-head review is complete with no findings. The SkillEval evidence limitation below remains explicit rather than being counted as a successful evaluation.

Checks performed:

  • Inspected all 87 changed paths by their source/generated roles. All 78 unrelated generated records change only source provenance. Source commit d5fc2121 is reachable and its canonical bytes are unchanged at this head; all 12 source-file SHA-256 values in the affected projection match fetched current-head bytes, and projected sidecar fields match their source.
  • Matched the 17 recorded upstream paths, byte counts and Git blob identities against the complete 213-entry pinned tree. Re-read the relevant timing/pair gate, queue/replay, profiler, aggregation, server response, quality and attribution sources, plus paper v3 §4. Recomputed the public 100+100 latency arrays and memory sample summaries. The guide preserves complete-request versus generated-action timing, per-task loss, precision differences, whole-device memory and missing measured-source/raw-episode evidence; it does not attribute the later SmolVLA experiment to paper v3.
  • The existing owner stays candidate / not_run, with no hardware prerequisite or new SearchSpace mechanism. Public sample arithmetic is not a new device measurement or an independent recount of unavailable episode logs.

Current hosted evidence: all six Ready jobs and the declaration check pass. The inspected Python 3.11 log reports Ruff success, 3,544 passed / 13 skipped / 284 subtests; the isolated wheel check reads 127 source-grounded records. Tested merge 954ff5c6 has zero file differences from this head. I did not rerun repository tests or execute upstream code.

The SkillEval plan selected zero comparisons and was not executed. Its unsupported executable snapshot entry is already present with the identical mode/blob on main; this review neither relaxes that safety boundary nor treats unavailable/unknown budget or evaluation as a pass. No runtime, robot, device, quality-preservation, performance or Agent-efficacy result is established. #339's overlapping pending amendment was not combined or tested here. This COMMENT is not formal APPROVE, merge authorization, or acceptance/closure of the broader #91/#177 goals.

@Dong1017 Dong1017 added the area:knowledge Knowledge line: packs, skill library, knowledge admission label Nov 4, 2026 — with ChatGPT Codex Connector

Copy link
Copy Markdown
Collaborator

Review at head 0f4433b7: no defect found in the new source record/reference in isolation (tools/build_skill_coverage.py --check passes locally), but I am not approving yet because of a cross-PR integration problem:

  1. Conflicts with the other open knowledge PRs: merging this head onto main together with any of Add source-bound TTS playout benchmark guidance #339, knowledge: audit Infera serving-search result provenance #332, knowledge: validate NInfer state and completion contracts #325, knowledge: audit DeepSeek C expert-cache byte validity #324 or knowledge: distinguish Xe-Forge trial origin and timing #322 produces 81-85 conflicting files in my local trial merges (generated repo_packs/.../knowledge/**, docs/skill-coverage.md, docs/reference-matrix.md, docs/searchspace-matrix.md). Add source-bound TTS playout benchmark guidance #339, knowledge: validate NInfer state and completion contracts #325 and knowledge: distinguish Xe-Forge trial origin and timing #322 also edit the same hand-written src/inferrsi/data/skill_library/methods/inference-benchmark-design/SKILL.md and metadata.json as this PR, so those need a real content merge, not only regeneration.
  2. Expected: please agree a merge order with the other authors; after each predecessor lands, rebase, re-merge inference-benchmark-design/SKILL.md/metadata.json, regenerate the projections from the new source revision and re-run the index/coverage checks, then request re-review on the new head (approvals would be dismissed by the push anyway).

Not verified by me: the upstream source pins/byte hashes in embodied-cpp-source.json against the upstream repository.

Copy link
Copy Markdown
Collaborator

Re-review at head 0f4433b7c9bbc1512a4ef5526f087c2a9730d239 (range 663f3b327998526a13aa31e72ad729025c18983c...0f4433b7c9bbc1512a4ef5526f087c2a9730d239). Not approving.

No new commits since the prior short-SHA note. Isolation content (Embodied.cpp VLA reference) still looks coherent, but blockers remain:

  1. Dirty vs main: mergeable_state=dirty; current main is 1246348f252dbc7e473efb1752c2d81da15e23fd (ahead of this PR's base). Rebase required.
  2. Cross-PR integration: sibling knowledge PRs still open (Add source-bound TTS playout benchmark guidance #339, knowledge: audit Infera serving-search result provenance #332, knowledge: validate NInfer state and completion contracts #325, knowledge: audit DeepSeek C expert-cache byte validity #324, knowledge: distinguish Xe-Forge trial origin and timing #322). Overlap on generated repo_packs/.../knowledge/** and matrix/coverage docs. Add source-bound TTS playout benchmark guidance #339/knowledge: validate NInfer state and completion contracts #325/knowledge: distinguish Xe-Forge trial origin and timing #322 also edit the same hand-written inference-benchmark-design/SKILL.md and metadata.json.
  3. Expected: agree a merge order; after each predecessor lands, rebase, re-merge SKILL/metadata, regenerate projections from the new source revision, re-run index/coverage checks, and request re-review on the new head.

Not verified by me: upstream source pins/byte hashes in embodied-cpp-source.json against the upstream repository.

Reviewed commit: 0f4433b7c9bbc1512a4ef5526f087c2a9730d239.

@Dong1017 Dong1017 added the kind:docs Documentation and template delivery label Nov 10, 2026
Merge main after the earlier source PRs and the shared source snapshot
landed. Where this PR and main extend the same files, keep main's additions
first, then the Embodied C++ additions. Generated projections keep main's bytes
until they are regenerated from this commit.

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com>
Run main's writers on the committed combined source. Only the edited
owner's record, the shared source snapshot and the coverage view change.

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com>
cursor[bot]
cursor Bot previously approved these changes Nov 10, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base bd61d20 and head f68c9f9 (full diff; this seed has no previous head).

Bounded VLA observation-to-action guidance on inference-benchmark-design: new readset and procedure, skill contract and failure modes, reference and search-space rows, and the generated coverage and knowledge-pack projection. The seeded prior note is Dong1017's source-only LGTM at 0f4433b7c9bbc1512a4ef5526f087c2a9730d239 (different base/head), not an open Dong-bot blocker.

Verified in the diff:

  • The 17 Embodied.cpp paths in metadata.json match the 17 files entries in references/embodied-cpp-source.json, pinned at 156bed151826780e2ce0132f8d0e3540889ae8cd.
  • Script SHA-256 2a7e94a4df662d2da41169cc2345b75e683c82e082fe590eba713302255e1a7b, artifact SHA-256 db53513cfdc52a03c3a4f2f1a2bfbab7a1eed45a9995873ac3bf34b7e9d7019d, and LICENSE SHA-256 88993458addec64e6922f552eb27ddd5f775d911db56205f167c885800011bf7 agree across the procedure, the source record, and the provenance sources entry.
  • Recorded summary arithmetic matches the stored fields: 308.0299474900005 / 50 is the C++ per-generated-action mean, the request ratio is 0.6276648803191476, and 1029/1671 is the sampled-memory ratio 0.6157989228007181. Success counts 166/200 and 172/200, with task 7 at 9/20 versus 20/20, stay attached to the author tables. Per-generated-action cost is amortization; control-rate and quality-equivalence conclusions stay withheld. Paper reuse stays section 4/Table 3, with the wider inspection recorded on the readset.
  • SKILL.md, metadata.json, and both applicability copies in the knowledge pack carry the same VLA scope, five failure modes, refusal clause, recommendation, and Embodied.cpp limitation. Coverage moves from 7 to 8 advice and from 13 to 15 readsets, matching the added recommendation and two provenance readsets. empirical_validation stays not_run. Search-space hardware stays source context; the existing N/A reason is unchanged.

Not verified here:

  • Upstream blobs, the 213-entry tree, arXiv v3, and the raw latency arrays were not re-fetched, so those blob identities and the sample recomputation were not independently reproduced.
  • python tools/build_skill_coverage.py --check was not run, and CI was not inspected.
  • Knowledge-pack source_revision 4c778ff95e77dfdcdd869f2e8a17d18d77cc7122 was not checked for reachability or byte identity against this head. The registry hash is unchanged, and skill-index.json is not in the diff.

No blocking finding in this scope. Author-reported sample arithmetic stays a published-array audit, separate from a local device measurement or efficacy result.

amy-why-3459
amy-why-3459 previously approved these changes Nov 10, 2026

@amy-why-3459 amy-why-3459 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base bd61d20c52b4d772a3ea282ae9cb3c0fd50075df (current main) and head f68c9f948875693bddb1da84699cbbcb7849beec. This is incremental from my last review at 0f4433b7c9bbc1512a4ef5526f087c2a9730d239, covering the main merge 4c778ff9 and the regeneration in f68c9f94.

  • Earlier blockers are resolved. The branch is up to date with main (0 behind), and all the sibling knowledge PRs I flagged (#339, #332, #325, #324, #322) have merged. Against main, the PR is now 9 files, all for the Embodied.cpp VLA reference.
  • The content merge kept the siblings' content. In inference-benchmark-design/metadata.json, every changed line is an existing entry extended with the VLA clause, and the NInfer, Faster CosyVoice and Xe-Forge content from the merged PRs is intact. In SKILL.md, only the description was reworded (adding VLA), plus a new VLA paragraph.
  • Generated records match the source. _provenance.json pins 4c778ff9. All 17 source_files hashes in knowledge/common/skills/inference-benchmark-design.json match the file bytes at both 4c778ff9 and the head. docs/skill-coverage.md has a new digest, and the matrix rows were added in place.
  • CI is green on this head. full-check (3.10 and 3.11), installed-package (3.10 and 3.11), pr-references, plan and publish pass. One earlier declarations run failed, and a later run on the same head passed.

Still unverified by me: the upstream source pins and byte hashes in embodied-cpp-source.json, which I didn't check against the upstream repository. Static review only, no local test run.

Merge main after the K01/K02 moves into src/inferrsi/knowledge. The owner
files follow their new location unchanged, the reference-matrix row follows
main's rows, and this PR's skill-library links point to the moved library.
Generated projections keep main's bytes until they are regenerated from this
commit.

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com>
Run main's writers on the committed combined source. Only the edited
owner's record, the shared source snapshot and the coverage view change.

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com>
@Dong1017
Dong1017 dismissed stale reviews from amy-why-3459 and cursor[bot] via 87c84d6 November 16, 2026 03:17

Copy link
Copy Markdown
Collaborator

Re-review of the new commits after the previously approved head f68c9f948875693bddb1da84699cbbcb7849beec was dismissed. Reviewed range f68c9f948875693bddb1da84699cbbcb7849beec...87c84d608a6e8d18c763d5f5d3e7495416d0e340 (merge 4b3ad9403c9f863421d1da2b2f2df13492582690 of main aaa17a5dd333449d15fb052bb09b2012d55f2327, then projection regeneration) against current main aaa17a5dd333449d15fb052bb09b2012d55f2327. Not approving yet.

Content of this head versus current main looks consistent with the previously approved tree after the K01/K02 path move:

  • git merge-tree --write-tree aaa17a5d 87c84d60 is conflict-free and equals this head's tree 032f895cbac3b5afb2e5bd54be8bd0372375d5b8; main is an ancestor (0 behind). GitHub currently reports mergeable_state=dirty with no test merge SHA, which I could not reproduce locally.
  • The four owner files (SKILL.md, metadata.json, and the two Embodied references) are byte-identical to f68c9f94 at the moved src/inferrsi/knowledge/data/skill_library/ paths.
  • Versus main, metadata.json is append-only (VLA clauses, tags, sources and readsets); SKILL.md only retitles the description and adds the VLA paragraph. The Embodied row/section was appended in docs/reference-matrix.md and docs/searchspace-matrix.md, and the six skill-library links point at existing new-path targets.
  • All 17 source_files hashes in the generated inference-benchmark-design.json match the file bytes at both 4b3ad940 and this head. _provenance.json is pinned to 4b3ad940. docs/skill-coverage.md updates the input digest and the owner description only.

Blocker:

  1. Ready-lane CI has not run on this head. CI — ready / review lane (full-check 3.10/3.11, installed-package 3.10/3.11, dashboard-browser, pr-references) has no workflow run for 87c84d608a6e8d18c763d5f5d3e7495416d0e340. The description treats those hosted checks as the test evidence; please trigger or approve the pull_request workflow (this PR is from the Dong1017/OmniRSI fork) and request re-review when it is green on this head. I did not run the index/coverage writers or the test suite locally.
    • declarations first failed on this SHA (run 43797316449) and later passed (run 43797474014) after the PR body was updated; plan/publish passed; execute was skipped.

Merge-order note (not the current approve blocker): open #456 also edits the same handwritten inference-benchmark-design SKILL.md/metadata.json plus the matrices and generated record. After one of these lands, the other will need a content merge, regeneration, and re-review. #382 and #386 also touch docs/skill-coverage.md (and older pack paths).

Still unverified by me: the upstream source pins and byte hashes in embodied-cpp-source.json against the upstream repository.

Reviewed commit: 87c84d608a6e8d18c763d5f5d3e7495416d0e340.

Merge main after docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis (InferMatrixLab#351).
Only the shared source snapshot and the coverage view overlapped; they keep
main's bytes until they are regenerated from this commit. This PR's
hand-written files are unchanged.

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com>
Run main's writers on the committed combined source. Only the edited
owner's record, the shared source snapshot and the coverage view change.

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com>

Copy link
Copy Markdown
Collaborator

Re-review of the new commits after 87c84d608a6e8d18c763d5f5d3e7495416d0e340. Reviewed range 87c84d608a6e8d18c763d5f5d3e7495416d0e340...352eb0b3639b92a7cf1935886acac3463aec7450 (merge 23d27be8bde499f8dadd1436657017cf2f4077ab of main f536288710c726a827bfa3e4357c2c911984cbc3, then projection regeneration) against current main f536288710c726a827bfa3e4357c2c911984cbc3. Not approving.

The merge/regeneration looks consistent with the previously reviewed tree:

  • Versus 87c84d60, the four owner files and the generated inference-benchmark-design.json are unchanged. _provenance.json is re-pinned to 23d27be8. docs/skill-coverage.md only updates the input digest and the performance-ceiling-analysis coverage row from docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351. The unique diff versus main remains the same 9 Embodied/VLA paths.
  • Main is an ancestor (0 behind). GitHub reports mergeable=true.
  • Ready-lane full-check (3.10 and 3.11), dashboard-browser, pr-references, plus plan/publish/declarations, all passed on this head. The earlier blocker that ready-lane had not been scheduled on 87c84d60 is resolved by this push.

Blocker:

  1. installed-package fails on this head. Both 3.10 and 3.11 fail in CI — ready / review lane run 43809203491:

    AssertionError: installed resource count is 927, expected 925

    at tests/check_installed.py:543. This PR adds two packaged references (embodied-cpp-boundary-and-quality.md and embodied-cpp-source.json) but does not update the inventory assert. Main after docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 already expects 925 for its two HuggingArch files; after the merge the combined tree contains both pairs, so the installed count is 927. Please bump the expected inventory to 927 and re-run the ready-lane installed-package jobs.

Still unverified by me: the upstream source pins and byte hashes in embodied-cpp-source.json against the upstream repository.

Reviewed commit: 352eb0b3639b92a7cf1935886acac3463aec7450.

The installed-package check pins the packaged resource count. This PR adds
two inference-benchmark-design references, so the count rises from 925 to
927, as InferMatrixLab#351 raised it for its own two references.

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com>
@Dong1017

Copy link
Copy Markdown
Collaborator Author

@amy-why-3459 @zengchuang-hw ready for re-review at 9081da21617f7222559589e6a84296fb5a29ab8f. The approvals on f68c9f94 were dismissed because main moved under it; the knowledge content is unchanged since that review.

  1. K01/K02 move. Main aaa17a5d moved the skill library into src/inferrsi/knowledge. Main had not changed inference-benchmark-design's SKILL.md or metadata.json, so this PR's four owner files land at the moved paths byte-identical to f68c9f94. The only hand-written conflict was the reference-matrix row, which follows main's rows. This PR's 6 skill-library links now use the moved path and resolve.
  2. docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351. Main f5362887 overlapped only on _provenance.json and docs/skill-coverage.md. Both were regenerated from 23d27be8 with main's writers, and both --check modes pass.
  3. Installed inventory. tests/check_installed.py now expects 927 installed resources (925 plus this PR's two references), as docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 raised it for its own two.

Against main the PR changes 10 paths: 6 canonical, 3 generated and tests/check_installed.py. git merge-tree --write-tree f5362887 9081da21 is conflict-free and equals the head tree.

CI at this head is green: full-check (3.10, 3.11), installed-package (3.10, 3.11), pr-references and dashboard-browser; plan, publish, declarations and knowledge-model-policy also pass.

Still not verified: the upstream source pins and byte hashes in embodied-cpp-source.json against the upstream repository.

@amy-why-3459 amy-why-3459 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base f536288710c726a827bfa3e4357c2c911984cbc3 (current main) and head 9081da21617f7222559589e6a84296fb5a29ab8f. This is incremental from my last review at 352eb0b3639b92a7cf1935886acac3463aec7450, covering the new commit 9081da21617f7222559589e6a84296fb5a29ab8f only.

  • Earlier blocker is resolved. tests/check_installed.py now expects 927 installed resources instead of 925, matching the two packaged Embodied.cpp references. Ready-lane installed-package (3.10 and 3.11) passed on this head in run 43811476687, together with full-check (3.10 and 3.11), dashboard-browser, and pr-references. plan/publish/declarations also passed.
  • The unique delta versus 352eb0b3 is that one-line inventory-count change; knowledge content is unchanged.

Still unverified by me: the upstream source pins and byte hashes in embodied-cpp-source.json against the upstream repository.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base f536288710c726a827bfa3e4357c2c911984cbc3 and head 9081da21617f7222559589e6a84296fb5a29ab8f.

Live hygiene: mergeable=true; prior triage dirty state on f68c9f94 is cleared (origin/main is an ancestor; merge-tree matches head tree 891bf47c). mergeable_state=blocked only for missing second current Approve (Amy already APPROVED this head; prior cursor Approve was dismissed after rebase).

Verified:

  • 10-path VLA observation-to-action guidance on inference-benchmark-design: SKILL/metadata/references, matrices, regenerated pack/coverage/provenance, installed inventory 925→927.
  • Metadata Embodied paths (17) match embodied-cpp-source.json; pack source_files SHA-256 match local SKILL/metadata/two new refs; provenance pin 23d27be8 is an ancestor of this head.
  • Published-sample arithmetic matches recorded fields (308.0299…/50, request ratio 0.62766…, memory 1029/1671); quality 166/200 vs 172/200 with task 7 9/20 vs 20/20 retained; SmolVLA follow-up kept separate from paper v3.
  • Spot-fetched upstream LICENSE, bench_vla_boundary.py, and fixed-boundary artifact at pin 156bed15 — bytes and SHA-256 match the source record.
  • python3 tools/build_skill_coverage.py --check exit 0; Ready lane green on this head (full-check 3.10/3.11, installed-package 3.10/3.11, pr-references, dashboard-browser, plan/publish/declarations, knowledge-model-policy). Owner stays candidate / not_run.

Not verified here: full 17-file upstream re-fetch / 213-entry tree; local build_skill_index.py --check blocked by host Git lacking --no-lazy-fetch (hosted Ready covers it). No merge.

No blocking finding in this scope.

@Dong1017
Dong1017 merged commit ba233f7 into InferMatrixLab:main Nov 16, 2026
12 checks passed
@Dong1017
Dong1017 deleted the codex/issue91-embodied-source branch November 16, 2026 07:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:knowledge Knowledge line: packs, skill library, knowledge admission kind:docs Documentation and template delivery

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants