You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add runtime-indexed prefill pruning for Qwen3-VL exports - #813
Reported validation: 45 targeted tests, Ruff/format; 20 full/pruned FP16 comparisons and 2,240 written KV-prefix pairs agree; CUDA ABI/capture/replay checked.
Known limit: padded captured/eager outputs differ and batched captured prefill fails, including unpruned controls. Full GPU CI is incomplete; capture compatibility is not fully passing.
Performance: Qwen3-VL-2B
Combined package/runtime results, not isolated gains from this PR. Production uses decoder INT4/group32; optimized uses 66 INT8 + 131 INT4 projections, FP16 vision/embeddings/KV, pruned prefill and decoder capture.
A100-SXM4-80GB
Historical production: Oct 28 (b679913a2310); optimized: Nov 5 frozen clean gate.
Tokens
Production TTFT ms
Optimized TTFT ms
Production decode tok/s
Optimized decode tok/s
Change
128
21.25
10.73
232.16
349.41
+50.50%
512
29.52
17.66
230.47
340.51
+47.75%
2048
62.12
47.78
219.57
325.15
+48.08%
4096
122.50
100.42
212.16
304.94
+43.73%
7680
241.71
206.10
202.21
288.68
+42.76%
H200
Baseline: official qwen3-vl-2b-instruct-cuda-gpu:2 on published ORT 1.31.1 / GenAI 0.19.0, CUDA 13. Optimized: retained Nov 8 FPA1 results, not rerun.
Limits: H200 is unpaired: prompts/harness differ; the new 448x448 image has 244 input tokens versus 356 historically. Ratios are descriptive, not causal speedups. A100 is not matched to the new H200 runtime; no cross-GPU hardware claim. Optimized ORT 5481f20b / GenAI 4e51307d + 41c96365 are frozen, not current PR-head benchmarks.
Review required: historical optimized export loses 9 AI2D answers versus the accepted native recipe; padded/batched capture limitations remain. No default promotion.
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution. 1 out of 2 committers have signed the CLA.
✅ hanbitmyths ❌ Copilot You have signed the CLA already but the status is still pending? Let us recheck it.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Pruning options are silently ignored by unsupported subclass build paths, and shared-weight decoder parity tests are missing to verify row selection and full-prompt cache preservation.
Adds opt-in runtime-indexed final-token projection to Qwen3-VL’s three-component export. The decoder selects hidden rows before the language-model head while preserving full-prompt attention and KV-cache processing; existing defaults remain unchanged.
Changes:
Thread pruning and runtime-index options through builders, tasks, and the decoder.
Add an explicit DeepStack opt-out and omit the runtime-reserved input from GenAI JSON mappings.
Document the runtime contract and add graph, option-validation, and row-selection tests.
File
Description
tests/build_graph/vision_language_test.py
Check pruning, runtime-input, and DeepStack graph contracts.
src/mobius/tasks/_vision_language_3model.py
Add Qwen-VL pruning options and model eligibility checks.
src/mobius/tasks/_base.py
Declare the optional decoder index input.
src/mobius/models/qwen_vl.py
Select hidden rows before the language-model head.
src/mobius/models/base.py
Support per-batch runtime indices in token selection.
src/mobius/models/_models_test.py
Test numerical row selection and unsupported task options.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The new test directly calls SerializeToString(), coupling it to the protobuf API instead of using ir.save or the existing OnnxModelSession wrapper.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Use IR serialization instead of direct protobuf serialization
src/mobius/models/_models_test.py:56
This calls SerializeToString() directly on the returned protobuf model, coupling the new test to the protobuf API. Please serialize with ir.save and pass the resulting filename to ORT instead, or use the existing OnnxModelSession wrapper, which already handles IR serialization.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The reserved-input ABI needs cross-stack validation of the rebased export/runtime pair because companion GenAI PR #2889 was inaccessible and the reported runtime gate predates the rebase.
Save the tiny gather fixture through onnx_ir instead of calling protobuf serialization directly. Preserve the existing two batch-row selection cases and repository conventions. Six targeted gather and decoder/cache parity cases pass; Ruff 0.16.2 lint and format checks are clean.
Signed-off-by: Sunghoon Choi <sunghcho@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fixed in 6cfb33ca: the runtime-index test now uses ir.save with a pytest temporary file instead of direct protobuf serialization. All 6 targeted gather/shared-weight decoder/cache tests passed, with no skips. Ruff 0.16.2 lint/format and whitespace checks passed. This follow-up changes only the test fixture, not production/performance code.
Fresh reserved-input ABI verification
This is new validation of the rebased pair, not reuse of the pre-rebase runtime gate:
Fresh full/unpruned and runtime-index-pruned FP16 Qwen3-VL-2B packages, built through the public Mobius APIs from raw revision 89644892e4d85e24eaac8bacfd4f463576704203, with the explicit DeepStack compatibility opt-out. No historical exports or graph surgery were substituted.
Matching Python 3.13 native bindings and the frozen six-library runtime bundle were path/hash checked. Tests ran on an A100-SXM4-80GB with eight CPU threads.
The exported reserved last_token_indices input is INT32 [B,1,1], absent from generic decoder JSON mappings; the unpruned graph has no reserved input. Native observations verified chunk-relative final non-PAD indices during prefill and zero during decode.
Comparison
Passing case pairs
Result
Full/pruned eager, batches 1-4; one-token, text, padded and real-image prompts
16
All nine logit states and per-row argmax values agree
Eager/captured batch 1; one-token, text and real-image prompts
3
Logits are bitwise equal
Full/pruned captured batch-1 padded control
1
All nine logit states and argmax values agree
Across these 20 comparisons, all 2,240 written KV-prefix tensor pairs were bitwise equal, covering all 28 layers at prefill and after eight forced decode steps. Full/pruned eager logits passed atol=0.02, rtol=0.001 (maximum absolute difference 0.03125, within the combined tolerance); the captured padded full/pruned maximum was 0.0078125. Unused cache-capacity tails were not compared. The pruned batch-1 capture profile recorded 4 actual captures / 33 graph launches.
Capture compatibility is not fully passing. The padded batch-1 captured result differs substantially from eager: maximum logit difference 12.28125, with initial argmax 11958 -> 576. The same mismatch reproduces with pruning disabled; pruned/unpruned captured outputs preserve the same result and written caches. This establishes independence from this PR's pruning, not the root cause or correctness of padded capture. Batch-2 captured text prefill also hits batch_size must be 1 when sequence_length > 1 and past context is given at both chunk sizes 16 and 2048. That ORT guard is unchanged upstream; its trigger has not been fully isolated. No passing full batched capture gate is claimed.
These FP16 ABI controls are not a new performance or MMLU/AI2D evaluation. The historical Foundry-only benchmark, pre-rebase scope warning, DeepStack limitation and additional nine-answer AI2D regression remain unchanged; promotion remains REVIEW REQUIRED.
Why CI was incomplete
The required PR L4/L5 run exceeded the 60-minute job limit. Both jobs were cancelled, with failures already present in their incomplete logs; these are not completed suite verdicts.
The unchanged-main all-model run also reached the time limit. 24 exact L4 failing node IDs overlap with the PR run.
A narrow BLOOM probe on unchanged main, head 1100038138f84eeb7bd59ccb9471c392deda4820, reproduced CUBLAS failure 3: the resource allocation failed; expr=cublasCreate(&cublas_handle_) before numerical parity could be evaluated. This does not prove VRAM exhaustion or that every partial failure has the same cause.
Shared task/exporter changes intentionally select run_all=true. The optional pinned real SmolLM lane is intentionally skipped because the workflow passes run_real: false; its fast CPU lane passed. Draft status is not the reason for that skip.
The baseline-failure investigation is bounded. No required check, selector or timeout was weakened, and neither the full GPU CI suite nor the fresh captured/eager compatibility gate is represented as green.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human confirmation of cross-runtime compatibility is needed because the reserved-input ABI depends on unreviewed GenAI changes, required L4/L5 CI was cancelled, and padded/batched capture validation remains unresolved.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Enabling pruning silently resets a supplied QwenVLTask’s runtime-index setting, causing right-padded batches to select the last physical row instead of the requested valid row.
Keep an existing QwenVLTask runtime-index setting when build_from_module enables pruning. Cover task/string flag combinations and actual right-padded batch prefill/decode/KV parity; document the option-preservation semantics.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
prune_prefill_prefix,runtime_last_token_indicesanddisable_deepstack; existing defaults and upstream DeepStack support are unchanged.last_token_indicesas INT32[B,1,1], outside generic decoder mappings; reject unsupported task-level options.Companions: GenAI #2889, Olive #2794, recipe #694.
Reported validation: 45 targeted tests, Ruff/format; 20 full/pruned FP16 comparisons and 2,240 written KV-prefix pairs agree; CUDA ABI/capture/replay checked.
Known limit: padded captured/eager outputs differ and batched captured prefill fails, including unpruned controls. Full GPU CI is incomplete; capture compatibility is not fully passing.
Performance: Qwen3-VL-2B
Combined package/runtime results, not isolated gains from this PR. Production uses decoder INT4/group32; optimized uses 66 INT8 + 131 INT4 projections, FP16 vision/embeddings/KV, pruned prefill and decoder capture.
A100-SXM4-80GB
Historical production: Oct 28 (
b679913a2310); optimized: Nov 5 frozen clean gate.H200
Baseline: official
qwen3-vl-2b-instruct-cuda-gpu:2on published ORT 1.31.1 / GenAI 0.19.0, CUDA 13. Optimized: retained Nov 8 FPA1 results, not rerun.Controls: batch 1, context 8192, FP16 KV, chunk 2048, 128 outputs, 8 inference threads, no prefix reuse. TTFT is HTTP first-token latency; decode uses 127 first-to-last intervals. A100: 7 production / 14 optimized samples. H200 baseline: 14 samples/row after warmups, GPU 1 / CPU cores 8-15; pre-rendered prompts, tokenization/image processing inside HTTP.
Limits: H200 is unpaired: prompts/harness differ; the new 448x448 image has 244 input tokens versus 356 historically. Ratios are descriptive, not causal speedups. A100 is not matched to the new H200 runtime; no cross-GPU hardware claim. Optimized ORT
5481f20b/ GenAI4e51307d+41c96365are frozen, not current PR-head benchmarks.Review required: historical optimized export loses 9 AI2D answers versus the accepted native recipe; padded/batched capture limitations remain. No default promotion.