Skip to content

task: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #348

Open

Description

@fake0fan

Current state: Existing Issue author @fake0fan has submitted PR #351, now 1694a0b, but this Issue still records Proposed, unassigned ownership and no accepted implementation delegation. The PR now contains the six content/coverage files plus 80 regenerated packaged knowledge files, and keeps Refs #348 rather than closure wording. At the prior 18c9a16 head, Ready had four software-check failures from stale packaged projections (including the reported 6 failed / 3538 passed full-check result). The 1694a0b5 incremental review clears the stale-index delta by static inspection; the subsequent current-head Ready result now has all six jobs passing, including both full-check and installed-package versions. This supersedes the earlier pending/full-check failure state, not the remaining review questions. The exact-head declaration now names claude-opus-5 and claude-opus-5-5 with effort unknown and extends its exception request to the generated records. The policy check still fails: explicit independent owner exception approval is absent, material AI author 1's model is unapproved, and author 2's effort is not allowed. The reviewer also retains the source/provenance-marker questions, blog-figure sourcing, and branch-only source-pin integration question; no merge strategy or new acceptance policy is chosen here. Source/index consistency and CI do not establish model/performance efficacy, acceptance or authority to close this Issue.
Decision needed: The requester/maintainers need to record the accepted scope/delegation and name the accountable and acceptance owners, which remain TBD below, including the existing gate-ownership/cross-link question. A submitted PR or reviewer's suggestions do not make those decisions.
Next action: Existing PR author @fake0fan retains the remaining source clarification and re-review work, accurate revision-specific documentation, and any exact-head policy-exception request to an authorized independent owner. The generated/packaged-consistency failure is no longer an outstanding repair at 1694a0b. Preserve the original proposed scope and acceptance text below; this status grants no implementation, policy-exception, merge or closure authority.

Goal and acceptance

methods/performance-ceiling-analysis L1 turns a model config plus a hardware
table entry into a runtime floor. It gates neither half of the demand side: the
skill has no procedure for checking that the weight and KV counts it divides by a
peak describe the model actually being deployed. safetensors appears nowhere
under this owner, while the mechanics already exist next door in
weight-loading-optimization
and a byte-level weight inventory already exists as an artifact pattern under
h3-c.

An L1 floor computed from a wrong parameter count is wrong in a way no roofline
check detects, because the roofline never looks at the model again. This issue
adds the missing prerequisite and records the external source it is derived from.

  • Problem / desired outcome: a reviewed-source prerequisite that anchors L1's
    weight demand against the checkpoint before any number derived from it is
    reported, with its blind spot stated rather than implied.
  • Completion criteria:
    • A reference document and a typed source-identity.json under
      methods/performance-ceiling-analysis/references/, recording the pinned
      commit, per-file git blob and sha256 digests, and the verbatim-vs-summary
      coverage of each inspected source.
    • A prerequisite step in SKILL.md covering two gates: the bidirectional
      per-tensor checkpoint join, and the per-rank weight ledger. The first needs no
      parallelism layout; passing it certifies nothing about per-rank bytes.
    • An explicit statement of what the weight anchor does not anchor (KV
      semantics: a windowed and a global attention with the same head dims have
      byte-identical weights).
    • A metadata.json source entry and a docs/reference-matrix.md row.
    • locally_verified stays false; empirical_validation stays not_run.

Scope and constraints

Ownership and authorization

  • Proposed by / actual requester: repository owner
  • Status: Proposed
  • Accountable parent (Issue-level owner, if assigned): TBD
  • Slice implementer(s) / acting contributor (if assigned): TBD
  • Goal-acceptance owner: TBD
  • Accepted decision or scoped delegation: Not yet granted

Inputs and deliverables

  • Relevant sources, revisions or linked issues:
    • github.com/shenh10/HuggingArch @ facbbbc96856e558b9084a94a5070b2e2c86cf0b
      (MIT; single init repo commit dated 2026-11-04T02:24:14Z), inspected
      2026-11-04. Readset: docs/validation.md, docs/measured_calibration.md,
      docs/evaluation_model.md, docs/known_limitations.md,
      backend/arch/validate/_matching.py, backend/arch/validate/_weight_ledger.py,
      backend/arch/validate/_ablation.py, backend/measure/scope.py,
      kernel_bench/emit.py.
    • A companion blog post dated 2026-10-08, read in part. The repository
      post-dates it by roughly four weeks and carries no history, so the post's
      figures are not attributable to this commit and are recorded as unverified
      where they are recorded at all.
    • Related: [RFC] Per-task SOL modeling and a KDA-flow gap metric for fused operators #303 (proposed; not a dependency of this issue).
  • Expected artifacts: one PR touching only the files named under completion
    criteria.

Completion and closure

  • Who accepts completion and may close: the named acceptance owner, not the PR merge.
  • Remaining uncertainty / next decision: whether the two gates belong to this
    owner alone or should be cross-declared as a reference from
    weight-loading-optimization; the PR proposes the former with a cross-link.

Activity

  1. amy-why-3459 commented on Nov 4, 2026

    @amy-why-3459
    Collaborator

    Thanks, the scoping here is clear, and separating "the weight anchor certifies X" from "it says nothing about KV semantics" is the right framing. A few points to settle before a PR:

    1. Where the gates live. weight-loading-optimization already owns the safetensors mechanics (references/fastsafetensors-transfer-readset.md). I agree with putting the correctness gate under performance-ceiling-analysis, since that's where the L1 number gets consumed, but please make the cross-link go both ways so the loading skill doesn't end up growing a second, slightly different checkpoint-inventory procedure.
    2. Reusing the existing inventory shape. repositories/h3-c/audit-h3-runtime-residency/references/minimax-h3-weight-inventory.json is already a byte-level inventory. Will the per-tensor join reuse that schema (or name the fields it shares with it), rather than adding a new one? One shape would make the two artifacts comparable.
    3. What the join compares. It would help to spell out the comparison key and tolerance: tensor name, dtype, shape, and byte size from the safetensors header against what the config implies, and how tied embeddings, quantization scales/zero-points, and MTP/draft heads are counted. Those are the usual places where config-derived parameter counts and checkpoint bytes disagree.
    4. Per-rank ledger inputs. Since the second gate needs a parallelism layout, please state which TP/EP/PP sharding rules it assumes (replicated vs. sharded tensors, expert placement) and that it stops if the layout isn't given, instead of falling back to a default.
    5. Source identity. Since HuggingArch has a single unversioned commit, recording git blob and sha256 per inspected file is good. I'd also keep the blog-post figures out of SKILL.md entirely, not just marked unverified, so they can't get quoted as numbers later.

    The out-of-scope list (no #303 bound model, no Class A/B vocabulary, locally_verified: false) looks right to me.

  2. added
    area:knowledgeKnowledge line: packs, skill library, knowledge admission
    kind:taskBounded delivery task with a delegated slice
    on Nov 4, 2026
  3. Dong1017 commented on Nov 16, 2026

    @Dong1017
    Collaborator

    Decision — Accept L1 weight-anchor scope (Dong-bot as @Dong1017 delegate)

    Recorded 2026-11-16. Authority: Dong-bot decision authority for @Dong1017 accept/decide. Thread basis: issue Goal/acceptance + Completion criteria; Amy scoping notes (6626759642); delivery PR #351 at head 126a0e6e447cf3184ef90a2dad06606325f33f8a (Amy APPROVED / PASS on that head).

    Accepted — bounded knowledge delivery

    Accept this issue’s stated Goal/acceptance and Completion criteria as the authorized delivery for anchoring L1 weight demand against the checkpoint under methods/performance-ceiling-analysis:

    • Reference + typed source-identity.json (pinned commit, per-file blob/sha256, coverage).
    • Two prerequisite gates in SKILL.md: bidirectional per-tensor checkpoint join; per-rank weight ledger (gate 1 alone certifies nothing per rank).
    • Explicit non-anchor for KV semantics.
    • metadata.json source entry + docs/reference-matrix.md row.
    • locally_verified: false; empirical_validation: not_run.
    • Out of scope stands: no [RFC] Per-task SOL modeling and a KDA-flow gap metric for fused operators #303 bound/Class A–B import; no measured corpus vendor; no new skill ID/registry; no measurement/E2E claim.

    Accepted — gate ownership / cross-link

    Correctness gates live under performance-ceiling-analysis (where L1 consumes the number). Keep a bidirectional cross-link with weight-loading-optimization so loading mechanics stay owned there and do not grow a second inventory procedure (Amy point 1).

    Confirmed — ownership (was TBD)

    Field Decision
    Accountable parent @Dong1017
    Slice implementer @fake0fan (acting via #351)
    Goal-acceptance owner @Dong1017 (closes this Issue when Completion criteria are met; PR merge alone does not close)
    Accepted delegation Scoped knowledge-content delivery matching the Completion criteria above; Refs #348 on the PR remains correct until the acceptance owner closes

    Still open (not granted by this accept)

    1. Merge of docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 — separate review/merge lane (needs second eligible Approve on the current head; this Issue accept does not merge the PR).
    2. Amy’s pre-PR comparison/ledger detail items (join key/tolerance, inventory schema reuse, per-rank layout stop rule, blog figures out of SKILL.md) — treat as delivery/review checklist on docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351; Amy’s current-head PASS is evidence they are satisfied for that head, not a substitute for acceptance-owner closure here.
    3. Branch-only source-pin / squash limitation — already documented; not waived here.

    Status

    Flipping status:proposed → status:accepted. Next action for delivery: finish #351 review/merge lane; acceptance owner closes #348 only after Completion criteria are verified on the merged revision.

  4. added
    status:acceptedThe decision owner accepted it in a recorded comment
    and removed
    status:proposedWaiting for the decision named in the status block
    on Nov 16, 2026
  5. Dong1017 commented on Nov 16, 2026

    @Dong1017
    Collaborator

    Post-merge acceptance check (#351 → main)

    Checked after squash-merge of #351 by @Dong1017 onto main at f536288710c726a827bfa3e4357c2c911984cbc3 (merged head content from 126a0e6e447cf3184ef90a2dad06606325f33f8a). Goal-acceptance owner: @Dong1017. PR wording was Refs #348 (merge alone does not close).

    Completion criteria on merged revision

    Criterion Status on f5362887
    Reference + typed source-identity.json under methods/performance-ceiling-analysis/references/ Met — huggingarch-checkpoint-anchor.md + huggingarch-source-identity.json (pin facbbbc9…, per-file blob/sha256, inspection coverage)
    Two prerequisite gates in SKILL.md (per-tensor join + per-rank ledger) Met — Evidence item 4 + Gate 1/2 wording
    Explicit non-anchor for KV semantics Met — weights-only / byte-identical KV blind spot stated in SKILL.md and the anchor
    metadata.json source entry + docs/reference-matrix.md row Met — HuggingArch + companion-post sources; matrix row present
    locally_verified: false; empirical_validation: not_run Met
    Out of scope (#303 bound/Class A–B, no new skill ID, no measurement claim) Held

    Remaining — blocks closure

    Bidirectional cross-link with weight-loading-optimization is incomplete.

    Accepted ownership decision (6905865813) required gates under performance-ceiling-analysis and a bidirectional cross-link so loading mechanics stay owned by WLO (Amy point 1).

    Until that reverse link lands on main, this Issue stays open. No other Issues are closed by #351 (Refs #348 only; #303 / #385 / #386 remain separate scope).

    Next

    Small follow-up PR (or amendment) adding the WLO → performance-ceiling-analysis / huggingarch-checkpoint-anchor cross-link; then re-check closure here. Branch-only source-pin / squash limitation stays documented and is not the closure blocker.

  6. Dong1017 commented on Nov 16, 2026

    @Dong1017
    Collaborator

    Next-step assignment (WLO → PCA)

    Follow-up to the post-merge check (6905998770) — no change to that gap analysis.

    Remaining acceptance gap (unchanged): WLO → PCA / huggingarch-checkpoint-anchor reverse cross-link under methods/weight-loading-optimization/ is still missing on main. Knowledge criteria from #351 are otherwise met.

    Who lands it: @fake0fan as slice implementer for the original delivery — please open a small follow-up PR that adds only the WLO → performance-ceiling-analysis / checkpoint-anchor pointer (SKILL.md, metadata.json, or a short reference note). Do not reopen scope beyond that reverse link.

    Closure: @Dong1017 (goal-acceptance owner) keeps #348 open until that reverse link is on main and re-checked here. Merge of #351 alone does not close.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:knowledgeKnowledge line: packs, skill library, knowledge admissionkind:taskBounded delivery task with a delegated slicestatus:acceptedThe decision owner accepted it in a recorded comment

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions