diff --git a/docs/skill-coverage.md b/docs/skill-coverage.md index a64217c89..fcc1c1d2f 100644 --- a/docs/skill-coverage.md +++ b/docs/skill-coverage.md @@ -4,7 +4,7 @@ Generated from the registry, the skill sidecars and the two maintained matrices. **Do not edit by hand**; regenerate after any registry, skill, matrix or frozen-plan change. -- Input digest: `sha256:cd71660769696a01153a9bce7a7b2165d02ef359c405086827f7b65425718f77` +- Input digest: `sha256:8f220788fd2d046f894d60991d7fa77c2b899705a88cd0970218f99fe73e276c` - Regenerate: `python tools/build_skill_coverage.py` - Verify without writing: `python tools/build_skill_coverage.py --check` @@ -55,11 +55,11 @@ optimization is fully covered or that any procedure has measured benefit. | Skill | Backend | Declared task | Applicability | Conditions | SearchSpace layer / stage | Reference Matrix | State | Gaps | |---|---|---|---|---|---|---|---|---| -| [ascend-aclgraph-capture-replay](../src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-aclgraph-capture-replay/SKILL.md) | ascend | Diagnose vLLM-Ascend ACLGraph capture/replay eligibility, stable-input and ordering contracts, and capture-resource ... | Diagnose vLLM-Ascend ACLGraph eligibility, capture/replay dispatch, caller-buffer ... | input_contract, produced_artifact, refusal_point
3 advice | 1 Accelerator · Diagnose / Configure / Optimize / Verify | 1 · Systems sources (vllm-project/vllm-ascend @ `8a8e1e05ecb64a30804c99c8f362909f9a0cb5bf` ...) | candidate · not_run · 1 readset(s) | no-eval | +| [ascend-aclgraph-capture-replay](../src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-aclgraph-capture-replay/SKILL.md) | ascend | Diagnose vLLM-Ascend ACLGraph capture/replay eligibility, stable-input and ordering contracts, and capture-resource ... | Diagnose vLLM-Ascend ACLGraph eligibility, batch descriptor matching, capture/replay ... | input_contract, produced_artifact, refusal_point
5 advice | 1 Accelerator · Diagnose / Configure / Optimize / Verify | 1 · Systems sources (vllm-project/vllm-ascend @ `8a8e1e05ecb64a30804c99c8f362909f9a0cb5bf` ...) | candidate · not_run · 1 readset(s) | no-eval | | [ascend-device-health](../src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-device-health/SKILL.md) | ascend | Pre-run health checks and recovery procedures for Ascend NPU devices on shared machines: card-group topology ... | Pre-run Ascend device-health, topology, occupancy and timing checks before a trusted ... | failure_modes, input_contract, produced_artifact
3 advice | 4 Network · Diagnose
5 OS · Observe / Profile / Diagnose / Configure / Verify | 1 · Systems sources (Ascend/MindIE-SD @ 8481d39) | candidate · not_run · 0 readset(s) | no-eval | | [ascend-distributed-parallel](../src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-distributed-parallel/SKILL.md) | ascend | Select and validate distributed parallelism strategy on Ascend NPU: topology-aware card-group selection, USP/CP/TP/PP ... | Select and diagnose Ascend parallel strategies and their activation/overlap, with a ... | failure_modes, input_contract, produced_artifact
4 advice | 1 Accelerator · Configure
4 Network · Diagnose / Configure / Optimize / Verify / Observe | 1 · Systems sources (Ascend/MindIE-SD @ 8481d39)
1 · Systems sources (ascend-ai-coding/awesome-ascend-skills @ 75ec89c)
1 · Systems sources (cann/hccl @ `96fab0a2acb761e7f8ab0f94aeb5138f200a345b` (CANN Open ...) | candidate · not_run · 3 readset(s) | no-eval | | [ascend-graph-mode](../src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-graph-mode/SKILL.md) | ascend | Diagnose an Ascend TorchAir ACL graph path from versioned configuration through FX compilation, capture-key selection ... | Diagnose the TorchAir Python ACL graph path from configuration and FX compilation through ... | failure_modes, input_contract, produced_artifact
3 advice | 1 Accelerator · Diagnose / Configure / Verify
3 SSD · Diagnose | 1 · Systems sources (Ascend/torchair @ `e1118e8b0fc1b95190ab53b732c0872989550e20` ...)
1 · Systems sources (ascend-ai-coding/awesome-ascend-skills @ 75ec89c) | candidate · not_run · 3 readset(s) | no-eval | -| [ascend-model-adaptation](../src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-model-adaptation/SKILL.md) | ascend | Patterns for porting LLM/VLM model adapters to Ascend NPU: FP8 checkpoint handling, KV replication with TP, multimodal ... | Checkpoint onboarding for LLM/VLM adapters on Ascend: weight/scale loading, KV-head ... | failure_modes, input_contract, produced_artifact
3 advice | 1 Accelerator · Configure
3 SSD · Diagnose / Configure | 1 · Systems sources (vllm-project/vllm-ascend @ bc8115ed) | candidate · not_run · 0 readset(s) | no-eval | +| [ascend-model-adaptation](../src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-model-adaptation/SKILL.md) | ascend | Patterns for porting LLM/VLM model adapters to Ascend NPU: FP8 checkpoint handling, KV replication with TP, multimodal ... | Checkpoint onboarding for LLM/VLM adapters on Ascend: weight/scale loading, KV-head ... | failure_modes, input_contract, produced_artifact, refusal_point
3 advice | 1 Accelerator · Configure
3 SSD · Diagnose / Configure | 1 · Systems sources (vllm-project/vllm-ascend @ bc8115ed) | candidate · not_run · 1 readset(s) | no-eval | | [ascend-operator-profiling](../src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-operator-profiling/SKILL.md) | ascend | Benchmark a single Ascend NPU operator using msprof op: build, collect with warmup, read the 8 CSV metrics files ... | Build, measure and diagnose one Ascend operator in isolation using msprof op, before ... | failure_modes, input_contract, produced_artifact
3 advice | 1 Accelerator · Observe / Profile
5 OS · Observe / Profile | 1 · Systems sources (ascend-ai-coding/awesome-ascend-skills @ 75ec89c) | candidate · not_run · 0 readset(s) | no-eval | | [ascend-operator-tuning](../src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-operator-tuning/SKILL.md) | ascend | Run a staged, accuracy-gated tuning campaign on a single Ascend NPU operator: set a numeric target up front, block ... | Run a bounded, accuracy-gated tuning campaign for a single Ascend operator already ... | failure_modes, input_contract, produced_artifact, refusal_point
3 advice | 1 Accelerator · Optimize | 1 · Systems sources (ascend-ai-coding/awesome-ascend-skills @ 75ec89c) | candidate · not_run · 0 readset(s) | no-eval | | [ascend-optimization-casebook](../src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-optimization-casebook/SKILL.md) | ascend | Casebook of NPU optimization cases for Ascend inference systems: symptom → diagnosis → action → evidence records for ... | Find source-derived prior cases for an already classified Ascend inference bottleneck ... | input_contract, produced_artifact, refusal_point
2 advice | 1 Accelerator · Optimize / Verify | 1 · Systems sources (Ascend/MindIE-SD @ 8481d39) | candidate · not_run · 0 readset(s) | no-eval | diff --git a/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json b/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json index 7ab45e60f..1b9f6d8e4 100644 --- a/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json +++ b/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json @@ -1,7 +1,7 @@ { "schema_version": 1, "repository": "https://github.com/InferMatrixLab/InferRSI", - "source_revision": "23d27be8bde499f8dadd1436657017cf2f4077ab", + "source_revision": "0143bc4e0228866a07cdc6c94e11fe0d76b93ab2", "registry": { "path": "src/inferrsi/knowledge/data/skill_library/skill-index.json", "sha256": "7904275c3383380db179287171e1430617aaa9ac4f439291eff8a71426987fbb" diff --git a/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/ascend/skills/ascend-aclgraph-capture-replay.json b/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/ascend/skills/ascend-aclgraph-capture-replay.json index 8fbfb6bae..4dbd175a4 100644 --- a/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/ascend/skills/ascend-aclgraph-capture-replay.json +++ b/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/ascend/skills/ascend-aclgraph-capture-replay.json @@ -13,7 +13,8 @@ "ascend", "vllm-ascend", "aclgraph", - "runtime" + "runtime", + "stream-synchronization" ], "origin": [ "external" @@ -24,8 +25,8 @@ }, "checkout_ref": "checkout:ascend-aclgraph-capture-replay", "source_files": { - "src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-aclgraph-capture-replay/SKILL.md": "d494fc7e539cbb7388189bd7b3f04bb5c4983fdd7a82484348aad1cee7f741f2", - "src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-aclgraph-capture-replay/metadata.json": "3ac69dd82ae798450092c830b62f4958dc2f5b1c7000f9c3f822f2b0693f8af4", + "src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-aclgraph-capture-replay/SKILL.md": "60dcf15b8d5678521d5650160b3404091a03fdc843ffe31a6e13a249b1f6a8be", + "src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-aclgraph-capture-replay/metadata.json": "255ffa4b9e7469d4b0e5c34cf9effff75b78413e9a1df87232d07d5d28212db6", "src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-aclgraph-capture-replay/references/source-notes.md": "6ffbfb68cc7e04923b4fbca2a73a7555b7c3b45a6f40819bb5f3213ef05cf88d" }, "skill_metadata": { @@ -35,7 +36,8 @@ "ascend", "vllm-ascend", "aclgraph", - "runtime" + "runtime", + "stream-synchronization" ], "status": "candidate", "empirical_validation": "not_run", @@ -54,27 +56,31 @@ ], "consumption_mode": "packaged", "applicability": { - "scope": "Diagnose vLLM-Ascend ACLGraph eligibility, capture/replay dispatch, caller-buffer ownership and capture-resource failures.", + "scope": "Diagnose vLLM-Ascend ACLGraph eligibility, batch descriptor matching, capture/replay dispatch, caller-buffer ownership, multi-stream ordering, and capture-resource failures.", "prerequisites": "An existing eager correctness baseline; authorization before any new run or live configuration change.", "input_contract": "Installed stack and device, requested/effective graph mode, runner/backend settings, capture sizes and batch descriptors, error phase, replay observations and HBM/stream evidence.", "produced_artifact": "A bounded diagnosis, smallest reversible proposal, expected tradeoff and correctness/resource verification plan with unresolved compatibility or authorization questions.", - "refusal_point": "Stop on output mismatch, unstable replay addresses, capture exhaustion, unknown ordering or version drift; missing evidence permits only a conditional proposal." + "refusal_point": "Stop on output mismatch, unstable replay addresses, capture exhaustion, stream resource errors without capture-size reduction, unknown ordering or version drift; missing evidence permits only a conditional proposal." }, "recommendations": [ "Diagnose the effective runtime mode and batch descriptor before changing capture sizes; intended warmup/profile bypass is not a cache defect.", + "Verify caller BatchDescriptor stability and dispatch mode before investigating graph replay issues.", + "For stream-resource exhaustion, reduce cudagraph_capture_sizes or max_cudagraph_capture_size.", "Trace stable replay buffers to the owning caller and preserve synchronization/update ordering; the wrapper does not stage new inputs into persistent buffers.", "Compare eager and graph execution on the same workload, separating initial capture, repeated replay, compiler changes and memory/stream costs; a replay log alone proves neither coverage nor speedup." ] }, "applicability": { - "scope": "Diagnose vLLM-Ascend ACLGraph eligibility, capture/replay dispatch, caller-buffer ownership and capture-resource failures.", + "scope": "Diagnose vLLM-Ascend ACLGraph eligibility, batch descriptor matching, capture/replay dispatch, caller-buffer ownership, multi-stream ordering, and capture-resource failures.", "prerequisites": "An existing eager correctness baseline; authorization before any new run or live configuration change.", "input_contract": "Installed stack and device, requested/effective graph mode, runner/backend settings, capture sizes and batch descriptors, error phase, replay observations and HBM/stream evidence.", "produced_artifact": "A bounded diagnosis, smallest reversible proposal, expected tradeoff and correctness/resource verification plan with unresolved compatibility or authorization questions.", - "refusal_point": "Stop on output mismatch, unstable replay addresses, capture exhaustion, unknown ordering or version drift; missing evidence permits only a conditional proposal." + "refusal_point": "Stop on output mismatch, unstable replay addresses, capture exhaustion, stream resource errors without capture-size reduction, unknown ordering or version drift; missing evidence permits only a conditional proposal." }, "recommendations": [ "Diagnose the effective runtime mode and batch descriptor before changing capture sizes; intended warmup/profile bypass is not a cache defect.", + "Verify caller BatchDescriptor stability and dispatch mode before investigating graph replay issues.", + "For stream-resource exhaustion, reduce cudagraph_capture_sizes or max_cudagraph_capture_size.", "Trace stable replay buffers to the owning caller and preserve synchronization/update ordering; the wrapper does not stage new inputs into persistent buffers.", "Compare eager and graph execution on the same workload, separating initial capture, repeated replay, compiler changes and memory/stream costs; a replay log alone proves neither coverage nor speedup." ], diff --git a/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/ascend/skills/ascend-model-adaptation.json b/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/ascend/skills/ascend-model-adaptation.json index cc9a1f0ee..13bfbb69e 100644 --- a/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/ascend/skills/ascend-model-adaptation.json +++ b/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/ascend/skills/ascend-model-adaptation.json @@ -14,7 +14,8 @@ "npu", "model-adaptation", "fp8", - "multimodal" + "multimodal", + "vllm-ascend" ], "origin": [ "external" @@ -25,8 +26,9 @@ }, "checkout_ref": "checkout:ascend-model-adaptation", "source_files": { - "src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-model-adaptation/SKILL.md": "8fdf46fb2743b67a66bc069d22a45305a222debfb606777bc451b4b8f31d9cc6", - "src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-model-adaptation/metadata.json": "989b91821b36c0b80caf1323ec527e7ac847219b8f4fe8ca4ea22d292365cc3c" + "src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-model-adaptation/SKILL.md": "ff1f0d941ac429041ca295871135e1b14e22c2819c404dd2435576c1d0cbc838", + "src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-model-adaptation/metadata.json": "e5191f402b0ccf069a0aafc9873b990704426f652c0048ad027082f5e47537dc", + "src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-model-adaptation/references/source-notes.md": "80df6181be3853190fd8fb35d4f853eb0056f4001071fe380dde0c3ef4ccde1b" }, "skill_metadata": { "name": "ascend-model-adaptation", @@ -36,7 +38,8 @@ "npu", "model-adaptation", "fp8", - "multimodal" + "multimodal", + "vllm-ascend" ], "status": "candidate", "profile": "local-skill-v1", @@ -50,14 +53,17 @@ "Source-derived candidate; no local efficacy validation.", "Original typed source inventory is not normalized; retain source identities and limits in the procedure/readsets." ], - "provenance_readsets": [], + "provenance_readsets": [ + "references/source-notes.md" + ], "consumption_mode": "packaged", "applicability": { "scope": "Checkpoint onboarding for LLM/VLM adapters on Ascend: weight/scale loading, KV-head replication under tensor parallelism and staged multimodal feature checks.", "prerequisites": "Check the installed CANN and checkpoint format against the versioned guidance; final sign-off requires real weights and at least one full-layer validation.", "input_contract": "Checkpoint weights/configuration and scale keys, attention-head/TP configuration, model modalities and expected graph/capacity features.", "produced_artifact": "A checkpoint/feature report separating dummy and real-weight evidence, applicable feature outcomes, and theoretical versus demonstrated serving capacity.", - "failure_modes": "Dummy-only success misses scale/remapping/sharding defects; blindly sharding replicated KV heads or accepting reduced-layer results can conceal loading or full-model failures." + "failure_modes": "Dummy-only success misses scale/remapping/sharding defects; blindly sharding replicated KV heads or accepting reduced-layer results can conceal loading or full-model failures.", + "refusal_point": "Stop on missing FP8 paired scale keys, unsupported tensor parallel KV-head split without replication, unverified multimodal visual encoder output, or failed base endpoint validation." }, "recommendations": [ "Gate onboarding in order: dummy architecture check, real-weight loading, model endpoint, text request and then multimodal request when the model supports VL; retain the evidence each stage actually supplies.", @@ -70,7 +76,8 @@ "prerequisites": "Check the installed CANN and checkpoint format against the versioned guidance; final sign-off requires real weights and at least one full-layer validation.", "input_contract": "Checkpoint weights/configuration and scale keys, attention-head/TP configuration, model modalities and expected graph/capacity features.", "produced_artifact": "A checkpoint/feature report separating dummy and real-weight evidence, applicable feature outcomes, and theoretical versus demonstrated serving capacity.", - "failure_modes": "Dummy-only success misses scale/remapping/sharding defects; blindly sharding replicated KV heads or accepting reduced-layer results can conceal loading or full-model failures." + "failure_modes": "Dummy-only success misses scale/remapping/sharding defects; blindly sharding replicated KV heads or accepting reduced-layer results can conceal loading or full-model failures.", + "refusal_point": "Stop on missing FP8 paired scale keys, unsupported tensor parallel KV-head split without replication, unverified multimodal visual encoder output, or failed base endpoint validation." }, "recommendations": [ "Gate onboarding in order: dummy architecture check, real-weight loading, model endpoint, text request and then multimodal request when the model supports VL; retain the evidence each stage actually supplies.", diff --git a/src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-aclgraph-capture-replay/SKILL.md b/src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-aclgraph-capture-replay/SKILL.md index be3d6ccbd..6aec75912 100644 --- a/src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-aclgraph-capture-replay/SKILL.md +++ b/src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-aclgraph-capture-replay/SKILL.md @@ -13,11 +13,14 @@ This candidate derives from `vllm-project/vllm-ascend@8a8e1e05ecb64a30804c99c8f3 Status: candidate; empirical validation: **not_run**. -## Related capabilities +## Related capabilities & responsibility routing -For checkpoint onboarding, FP8 loading, KV-head replication and multimodal validation order, use the older documentation-based [ascend-model-adaptation](../ascend-model-adaptation/SKILL.md). Its startup guidance is adjacent to this wrapper contract, not evidence that the two source revisions form a validated stack. See the [version and responsibility table](../../../README.md#vllm-ascend-version-and-responsibility-routing) for their separate evidence and date boundaries. +For checkpoint onboarding, FP8 loading, KV-head replication and multimodal validation order, use [ascend-model-adaptation](../ascend-model-adaptation/SKILL.md). Its startup guidance is adjacent to this wrapper contract, not evidence that the two source revisions form a validated stack. See the [version and responsibility table](../../../README.md#vllm-ascend-version-and-responsibility-routing) for their separate evidence and date boundaries. -[ascend-graph-mode](../ascend-graph-mode/SKILL.md) covers TorchAir configuration guidance; this skill adds the independently pinned vLLM-Ascend runtime wrapper. [cann-attribution-overlap](../cann-attribution-overlap/SKILL.md) supplies attribution methodology, not evidence that this wrapper improved performance. [torchnpu-device-memory-lifecycle](../torchnpu-device-memory-lifecycle/SKILL.md) addresses allocator ownership at another pin, not graph-pool caller correctness. +- **Runtime Wrapper (`ascend-aclgraph-capture-replay`)**: Handles graph pool capture, batch descriptor matching, input pointer assertions, and replay dispatch. +- **Compiler Passes (`ascend-graph-mode`)**: Covers TorchAir configuration guidance; this skill adds the independently pinned vLLM-Ascend runtime wrapper. +- **Attribution Discipline (`cann-attribution-overlap`)**: Supplies attribution methodology, not evidence that this wrapper improved performance. +- **Device Allocator (`torchnpu-device-memory-lifecycle`)**: Addresses allocator ownership at another pin, not graph-pool caller correctness or stream lifetimes. ## Required evidence (P) @@ -29,9 +32,18 @@ If evidence is unavailable, identify the missing item and give a conditional pro 1. **Confirm the execution path.** Compare the installed implementation with the pin. Separate compile-time Npugraph_ex optimization from runtime ACLGraph. The guide supports V1/V2, but warns that context-parallel `FULL` is insufficiently supported. Backend compatibility may narrow the requested mode. Diagnose the effective mode; do not force a requested mode through compatibility checks. [F1] 2. **Classify dispatch before tuning.** Read the forward context's runtime mode and batch descriptor. `NONE` or a wrapper-mode mismatch calls the runnable directly. A matching mode captures an uncaptured entry and replays a captured entry. Treat intended warmup/profile bypass as expected; investigate changing descriptors or failed captures before calling repeated work a cache defect. [F2] -3. **Check replay correctness first.** Require caller-managed stable replay buffers. The wrapper neither owns persistent input buffers nor copies new inputs into them. Propose `VLLM_LOGGING_LEVEL=DEBUG` before wrapper creation for a diagnostic reproduction: its address assertion covers only top-level positional tensor arguments, not keyword/nested inputs or semantic correctness. Escalate an address mismatch to the buffer-owning caller; never suppress it to obtain a timing. [F3] -4. **Preserve lifetime and ordering.** Review offloader synchronization/join, graph-pool ownership, output/workspace weak references, and the FULL replay/update branch before proposing code changes. Do not remove synchronization, reorder update/replay, or generalize weak-output handling as a performance shortcut. Follow the branch-specific invariants in source-notes. [F4–F5] -5. **Choose one evidence-matched proposal.** For confirmed stream-resource capture failures, propose fewer capture sizes or a smaller maximum. Consider `FULL`/`FULL_DECODE_ONLY` only for mostly uniform decode with verified backend/platform support; this can also change compilation behavior. Use temporary eager execution as a diagnostic control. For the specific old-HDK signature, report the source's HDK 25.5.1+ and matching-CANN guidance as signature-specific, not a universal minimum. Preserve unrelated exceptions for diagnosis. [F1, F6] +3. **Classify batch descriptor matching and capture sizing.** The wrapper indexes captured graphs by `BatchDescriptor`. A matching runtime mode with a missing entry triggers initial capture (after `validate_cudagraph_capturing_enabled()`), while `NONE` or a mode mismatch bypasses capture/replay and calls the runnable directly [F2]. Capture sets are bounded by `cudagraph_capture_sizes` and `max_cudagraph_capture_size` to prevent stream-resource exhaustion; the guide notes larger capture sets increase resource pressure [F6]. Any request bucketing or padding to stabilize batch descriptors operates upstream in the caller/runner, not inside this wrapper. +4. **Check replay correctness first.** Require caller-managed stable replay buffers. The wrapper neither owns persistent input buffers nor copies new inputs into them. Propose `VLLM_LOGGING_LEVEL=DEBUG` before wrapper creation for a diagnostic reproduction: its address assertion covers only top-level positional tensor arguments (`args`), not keyword arguments (`kwargs`), nested tensor containers, or semantic correctness [F3]. Equal addresses alone establish neither updated contents nor semantic validity. Escalate an address mismatch to the buffer-owning caller; never suppress it to obtain a timing. +5. **Preserve multi-stream synchronization and ordering.** Note that the relative ordering of parameter updates and replay does not apply unconditionally across all dispatch paths; it is specific to the `FULL` graph mode on update-aware paths where `use_updatable_graph(attn_backend)` holds [F5]: + - On this update-aware `FULL` path when ENPU is enabled: parameter update runs before graph replay (`update` then `replay`). The pinned source notes this ENPU ordering is assigned to the model runner caller, which was not inspected. + - On this update-aware `FULL` path when ENPU is disabled: graph replay executes first, followed by parameter update (`replay` then `update`). + - `FULL` synchronizes the current stream when ENPU is disabled, except in the draft-model-plus-EAGLE case. + - For piecewise or non-updatable graph paths, verify stream ownership and event dependencies directly against `vllm_ascend/compilation/acl_graph.py` before inferring synchronization order. + - `weak_ref_output` is safe only for the last piecewise graph whose output is not consumed downstream. Capture returns local `output`, whereas replay returns `entry.output`. Follow the branch-specific invariants in source-notes. [F4–F5] +6. **Choose one evidence-matched proposal.** + - For confirmed stream-resource capture failures (`_is_stream_resource_capture_error`: `insufficient_stream_resources`, `stream resources are insufficient`, or code `207008` paired with `stream resource`), propose fewer capture sizes or a smaller maximum (`max_cudagraph_capture_size`). Consider `FULL`/`FULL_DECODE_ONLY` for mostly uniform decode with verified backend support. + - For old HDK failures (`_is_old_hdk_capture_error`: `alloc sq cq fail`), report the source's HDK 25.5.1+ and matching-CANN guidance as signature-specific, not a universal minimum. + - Use temporary eager execution (`--enforce-eager`) as a diagnostic control. Preserve unrelated exceptions for diagnosis. [F1, F6] ## Benefit, cost, and counter-signals (P) diff --git a/src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-aclgraph-capture-replay/metadata.json b/src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-aclgraph-capture-replay/metadata.json index 2aaea9993..bd58f81b4 100644 --- a/src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-aclgraph-capture-replay/metadata.json +++ b/src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-aclgraph-capture-replay/metadata.json @@ -3,7 +3,8 @@ "ascend", "vllm-ascend", "aclgraph", - "runtime" + "runtime", + "stream-synchronization" ], "status": "candidate", "empirical_validation": "not_run", @@ -22,14 +23,16 @@ ], "consumption_mode": "packaged", "applicability": { - "scope": "Diagnose vLLM-Ascend ACLGraph eligibility, capture/replay dispatch, caller-buffer ownership and capture-resource failures.", + "scope": "Diagnose vLLM-Ascend ACLGraph eligibility, batch descriptor matching, capture/replay dispatch, caller-buffer ownership, multi-stream ordering, and capture-resource failures.", "prerequisites": "An existing eager correctness baseline; authorization before any new run or live configuration change.", "input_contract": "Installed stack and device, requested/effective graph mode, runner/backend settings, capture sizes and batch descriptors, error phase, replay observations and HBM/stream evidence.", "produced_artifact": "A bounded diagnosis, smallest reversible proposal, expected tradeoff and correctness/resource verification plan with unresolved compatibility or authorization questions.", - "refusal_point": "Stop on output mismatch, unstable replay addresses, capture exhaustion, unknown ordering or version drift; missing evidence permits only a conditional proposal." + "refusal_point": "Stop on output mismatch, unstable replay addresses, capture exhaustion, stream resource errors without capture-size reduction, unknown ordering or version drift; missing evidence permits only a conditional proposal." }, "recommendations": [ "Diagnose the effective runtime mode and batch descriptor before changing capture sizes; intended warmup/profile bypass is not a cache defect.", + "Verify caller BatchDescriptor stability and dispatch mode before investigating graph replay issues.", + "For stream-resource exhaustion, reduce cudagraph_capture_sizes or max_cudagraph_capture_size.", "Trace stable replay buffers to the owning caller and preserve synchronization/update ordering; the wrapper does not stage new inputs into persistent buffers.", "Compare eager and graph execution on the same workload, separating initial capture, repeated replay, compiler changes and memory/stream costs; a replay log alone proves neither coverage nor speedup." ] diff --git a/src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-model-adaptation/SKILL.md b/src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-model-adaptation/SKILL.md index b51c0bc66..cac67352d 100644 --- a/src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-model-adaptation/SKILL.md +++ b/src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-model-adaptation/SKILL.md @@ -7,15 +7,18 @@ description: 'Patterns for porting LLM/VLM model adapters to Ascend NPU: FP8 che # Ascend Model Adaptation -Status: candidate. Source-grounded from vllm-ascend @ bc8115ed, -`.agents/skills/vllm-ascend-model-adapter/references/`. Unverified locally -(no NPU). Verify against your CANN version and checkpoint format. +Status: candidate. Source-grounded from vllm-ascend @ `bc8115ed7fdc7e500320b5de70742ef049b4c1a2`. Read [source-notes](references/source-notes.md) for immutable references and attribution boundaries. Unverified locally (no NPU). Verify against your CANN version and checkpoint format. -## Choose the versioned owner +## Choose the versioned owner & three-tier responsibility -This entry retains the historical model-adapter documentation pin `bc8115ed`. Use it for checkpoint onboarding, FP8 loading, KV-head replication and multimodal validation order. For the separately pinned runtime capture/replay contract, follow [ascend-aclgraph-capture-replay](../ascend-aclgraph-capture-replay/SKILL.md). Its wrapper evidence does not revalidate this entry or establish a compatible installed stack. +This entry retains the historical model-adapter documentation pin `bc8115ed`. Use it for checkpoint onboarding, FP8 loading, KV-head replication and multimodal validation order. -The [version and responsibility table](../../../README.md#vllm-ascend-version-and-responsibility-routing) preserves each source scope, existing evidence inventory and original inspection-date boundary. +To prevent duplicate definitions and maintain strict attribution boundaries across repository owners, InferRSI adopts the following routing convention: +1. **Tier 1 — vLLM Generic Mechanism (`repositories/vllm/`)**: PagedAttention block allocation, KV cache indexing, preemption scheduling, worker RPC protocols. Does not assume NPU or CANN specifics. +2. **Tier 2 — vLLM Ascend Plugin Features (`hardware/ascend/` & `hardware/host/`)**: Model checkpoint weight mapping, FP8 load-time dequantization into BF16, TP-aware KV-head replication, runtime ACLGraph model runner, and native host KV offload descriptor pooling. +3. **Tier 3 — Ascend / CANN / TorchNPU Bottom Constraints (`hardware/ascend/`)**: Lower-level driver and device constraints such as `aclmdlExecute`, geExecutor, HCCL mesh scratch, TorchNPU caching allocator, H2D/D2H copy streams. + +For the runtime capture/replay execution contract, follow [ascend-aclgraph-capture-replay](../ascend-aclgraph-capture-replay/SKILL.md). Its wrapper evidence does not revalidate this entry or establish a compatible installed stack. For compiler pass options, follow [ascend-graph-mode](../ascend-graph-mode/SKILL.md). For host offload descriptor pooling, follow [host-memory-management](../../host/host-memory-management/SKILL.md). See the [version and responsibility table](../../../README.md#vllm-ascend-version-and-responsibility-routing) for their separate evidence boundaries. ## Debug order for a new checkpoint @@ -37,12 +40,12 @@ Source: `fp8-on-npu-lessons.md` § reporting-discipline. ## FP8 checkpoints on Ascend FP8 execution kernels are typically not supported on Ascend NPU at the operator -level. The recommended path is to dequantize FP8 weights to BF16 during loading. +level in baseline stacks. The recommended path is to dequantize FP8 weights to BF16 during loading. -Pattern: -- For each quantized weight, look for paired tensors: `.weight` and - `.weight_scale_inv`. -- Dequantize at load time: `weight = weight_fp8 * weight_scale_inv`. +Pattern (Source facts vs. implementation interpretation): +- **Source Fact**: For each quantized weight, look for paired tensors: `.weight` and + `.weight_scale_inv`. Upstream notes establish that load-time dequantization into BF16 is required when native FP8 execution kernels are unavailable on NPU, and enforce strict unpaired scale/weight checks where a missing scale key triggers loader failure. +- **Implementation Interpretation**: The arithmetic expression `weight = weight_fp8.to(torch.bfloat16) * weight_scale_inv` represents an implementation interpretation for checkpoints using reciprocal scale factors. Upstream notes do not define a single universal scale granularity: for block-quantized or per-channel checkpoints, shape handling, broadcasting, and expansion are checkpoint-specific. - Keep strict unpaired scale/weight checks — a missing scale key causes silent corruption if not caught. @@ -58,7 +61,7 @@ replicated across TP ranks. This requires a distinct loading path. Symptom: shape mismatch such as `128 vs 64` during model loading with TP. Pattern: -- Detect KV-head replication explicitly: if `tp_size > num_kv_heads`, load each +- Detect KV-head replication explicitly: if `tp_size > num_key_value_heads`, load each KV head into all ranks that share it rather than sharding it. - Use a local norm/shard loader for replicated KV heads. - Do not assume uniform divisibility for all head dimensions — check each @@ -78,7 +81,7 @@ Startup behavior with graph mode enabled: For stability: prefer `HCCL_OP_EXPANSION_MODE=AIV` with graph mode enabled. Use `--enforce-eager` only as a temporary isolation fallback during debugging, -not as a default serving configuration. +not as a default serving configuration. For the complete runtime wrapper dispatch contract, refer to [ascend-aclgraph-capture-replay](../ascend-aclgraph-capture-replay/SKILL.md). Source: `fp8-on-npu-lessons.md` § 5, `multimodal-ep-aclgraph-lessons.md` § 3. ## Multimodal feature validation order diff --git a/src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-model-adaptation/metadata.json b/src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-model-adaptation/metadata.json index 767cd2646..c8fcb9eb0 100644 --- a/src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-model-adaptation/metadata.json +++ b/src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-model-adaptation/metadata.json @@ -4,7 +4,8 @@ "npu", "model-adaptation", "fp8", - "multimodal" + "multimodal", + "vllm-ascend" ], "status": "candidate", "profile": "local-skill-v1", @@ -18,14 +19,17 @@ "Source-derived candidate; no local efficacy validation.", "Original typed source inventory is not normalized; retain source identities and limits in the procedure/readsets." ], - "provenance_readsets": [], + "provenance_readsets": [ + "references/source-notes.md" + ], "consumption_mode": "packaged", "applicability": { "scope": "Checkpoint onboarding for LLM/VLM adapters on Ascend: weight/scale loading, KV-head replication under tensor parallelism and staged multimodal feature checks.", "prerequisites": "Check the installed CANN and checkpoint format against the versioned guidance; final sign-off requires real weights and at least one full-layer validation.", "input_contract": "Checkpoint weights/configuration and scale keys, attention-head/TP configuration, model modalities and expected graph/capacity features.", "produced_artifact": "A checkpoint/feature report separating dummy and real-weight evidence, applicable feature outcomes, and theoretical versus demonstrated serving capacity.", - "failure_modes": "Dummy-only success misses scale/remapping/sharding defects; blindly sharding replicated KV heads or accepting reduced-layer results can conceal loading or full-model failures." + "failure_modes": "Dummy-only success misses scale/remapping/sharding defects; blindly sharding replicated KV heads or accepting reduced-layer results can conceal loading or full-model failures.", + "refusal_point": "Stop on missing FP8 paired scale keys, unsupported tensor parallel KV-head split without replication, unverified multimodal visual encoder output, or failed base endpoint validation." }, "recommendations": [ "Gate onboarding in order: dummy architecture check, real-weight loading, model endpoint, text request and then multimodal request when the model supports VL; retain the evidence each stage actually supplies.", diff --git a/src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-model-adaptation/references/source-notes.md b/src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-model-adaptation/references/source-notes.md new file mode 100644 index 000000000..c6a7f83c7 --- /dev/null +++ b/src/inferrsi/knowledge/data/skill_library/hardware/ascend/ascend-model-adaptation/references/source-notes.md @@ -0,0 +1,34 @@ +# Source Notes: Ascend Model Adaptation + +Evidence origin: Historical model-adapter documentation bundle `.agents/skills/vllm-ascend-model-adapter/references/`. + +## Provenance and limits + +- Repository: `vllm-project/vllm-ascend`. +- Immutable inspected commit: `bc8115ed7fdc7e500320b5de70742ef049b4c1a2` (resolves the historical prefix `bc8115ed` recorded in `src/inferrsi/knowledge/data/skill_library/README.md#vllm-ascend-version-and-responsibility-routing` against upstream `vllm-project/vllm-ascend`). +- Documentation sources: + - `fp8-on-npu-lessons.md`: FP8 paired weight loading, load-time dequantization to BF16, KV replication under Tensor Parallelism when `tp_size > num_key_value_heads`, and layer-reduction smoke testing. + - `multimodal-ep-aclgraph-lessons.md`: Staged multimodal feature validation order, MoE-only EP/flashcomm1 applicability rules, and startup graph-mode verification. +- Scope: Model adapter onboarding and checkpoint format handling on Ascend NPU. This skill does not cover runtime ACLGraph execution dispatch (see `ascend-aclgraph-capture-replay`), TorchAir compiler configuration (see `ascend-graph-mode`), or lower-level device allocator behavior (see `torchnpu-device-memory-lifecycle`). + +## Three-Tier Responsibility (InferRSI Routing Convention) + +To organize knowledge boundaries and prevent duplicate definitions across skills, InferRSI adopts the following routing convention: +1. **Tier 1 — vLLM Generic Mechanism (`repositories/vllm/`)**: + - PagedAttention block abstraction, KV cache indexing, scheduler preemption, worker RPC protocol. + - Independent of Ascend or NPU specifics. +2. **Tier 2 — vLLM Ascend Plugin Features (`hardware/ascend/` & `hardware/host/`)**: + - Model adapter checkpoint weight conversion, FP8 load-time dequantization into BF16, TP-aware GQA/MQA KV replication, ACLGraph runtime runner, and native host KV offload descriptor pooling. +3. **Tier 3 — Ascend / CANN / TorchNPU Bottom Constraints (`hardware/ascend/`)**: + - Lower-level CANN runtime, `aclmdlExecute`, geExecutor, HCCL mesh scratch, TorchNPU caching allocator, and H2D/D2H copy streams. + +## Inspected source facts + +- **F1 (Onboarding Gate Sequence):** `--load-format dummy` checks operator coverage, imports, and shape signatures without weight loading. Real-weight loading validates tensor mapping, FP8 paired-tensor presence, and sharding. Single-text and multimodal endpoints must pass before capacity testing. +- **F2 (FP8 Load-time Dequantization):** NPU execution kernels typically lack native FP8 operator dispatch. Upstream notes record that FP8 weights must be dequantized to BF16 at load time using paired tensors (`.weight` and `.weight_scale_inv`). Missing scale keys in checkpoints trigger failure; they are not framework errors. The formula `weight_bf16 = weight_fp8.to(torch.bfloat16) * weight_scale_inv` is an operational implementation interpretation for reciprocal scales, not a verbatim upstream equation. Upstream notes do not specify a single universal scale granularity: for block-quantized checkpoints, shape handling and broadcasting/expansion are checkpoint-specific. +- **F3 (KV-Head Replication under TP):** When tensor parallelism size exceeds key-value heads (`tp_size > num_key_value_heads`), KV heads must be replicated across ranks sharing them rather than sharded, avoiding dimension mismatches (e.g. 128 vs 64). +- **F4 (Multimodal Staged Feature Checks):** Text completion -> image+text completion -> graph mode logs (`Replaying aclgraph`) -> target context length / batch capacity. MoE-only features (EP, flashcomm1) are marked `N/A` for dense architectures. + +## Operational interpretation + +The procedure converts F1–F4 into a gated debugging checklist. It distinguishes verbatim source facts (F1–F4) from implementation interpretations (e.g. the specific multiplication formula for reciprocal FP8 scales) and InferRSI routing conventions (the three-tier knowledge boundary). All recommendations remain candidates; no local hardware execution or empirical speedup is claimed. diff --git a/tests/test_vllm_ascend_distill_skills.py b/tests/test_vllm_ascend_distill_skills.py new file mode 100644 index 000000000..3a8941e13 --- /dev/null +++ b/tests/test_vllm_ascend_distill_skills.py @@ -0,0 +1,82 @@ +from pathlib import Path + +import pytest + +from inferrsi.knowledge.catalog import search_knowledge +from inferrsi.knowledge.skill_catalog import skill_instructions, skill_metadata, skill_resource + +LIBRARY = Path(__file__).resolve().parents[1] / "src/inferrsi/knowledge/data/skill_library" + + +@pytest.mark.parametrize( + ("query", "expected_skill_id"), + [ + # Query 1: ACLGraph capture/replay runtime contract + ( + "ACLGraph capture replay", + "inferrsi.ascend.skills.ascend_aclgraph_capture_replay", + ), + ( + "vLLM-Ascend ACLGraph", + "inferrsi.ascend.skills.ascend_aclgraph_capture_replay", + ), + # Query 2: Model adaptation and FP8 checkpoint onboarding + ( + "FP8 checkpoint", + "inferrsi.ascend.skills.ascend_model_adaptation", + ), + # Query 3: KV replication with Tensor Parallelism + ( + "KV replication TP", + "inferrsi.ascend.skills.ascend_model_adaptation", + ), + # Query 4: Multimodal feature validation + ( + "multimodal feature validation", + "inferrsi.ascend.skills.ascend_model_adaptation", + ), + ], +) +def test_batch1_skills_are_discoverable_for_representative_queries( + query, expected_skill_id +): + results = search_knowledge(query, repo="inferrsi", backend="ascend", limit=5) + skill_ids = [item["id"] for item in results] + assert expected_skill_id in skill_ids, ( + f"Query {query!r} expected {expected_skill_id} in {skill_ids}" + ) + + +def test_aclgraph_skill_metadata_and_contracts(): + meta = skill_metadata(LIBRARY, "ascend-aclgraph-capture-replay") + assert meta["status"] == "candidate" + assert meta["empirical_validation"] == "not_run" + assert "references/source-notes.md" in meta["provenance_readsets"] + + notes = skill_resource( + LIBRARY, "ascend-aclgraph-capture-replay", "references/source-notes.md" + ).read_text(encoding="utf-8") + assert "8a8e1e05ecb64a30804c99c8f362909f9a0cb5bf" in notes + assert "acl_graph.py" in notes + + inst = skill_instructions(LIBRARY, "ascend-aclgraph-capture-replay") + assert "cudagraph_capture_sizes" in inst + assert "ENPU" in inst + + +def test_model_adaptation_skill_metadata_and_contracts(): + meta = skill_metadata(LIBRARY, "ascend-model-adaptation") + assert meta["status"] == "candidate" + assert meta["empirical_validation"] == "not_run" + assert "references/source-notes.md" in meta["provenance_readsets"] + + notes = skill_resource( + LIBRARY, "ascend-model-adaptation", "references/source-notes.md" + ).read_text(encoding="utf-8") + assert "bc8115ed7fdc7e500320b5de70742ef049b4c1a2" in notes + assert "Three-Tier Responsibility" in notes + + inst = skill_instructions(LIBRARY, "ascend-model-adaptation") + assert "--load-format dummy" in inst + assert "weight_scale_inv" in inst + assert "tp_size > num_key_value_heads" in inst