Repository navigation
feat(gguf): serve image input on qwen3vl ggufs from a deepstack qwen3vl_merger mmproj - #827
Merged
jason-fxz merged 1 commit intoOct 26, 2026
Conversation
void-mckenzie
marked this pull request as ready for review
October 26, 2026 20:11
Collaborator
|
Cross-checked at 54bb59f on an RTX PRO 6000 Blackwell (96 GB) against a local implementation of the same change:
LGTM. |
This was referenced Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #671 (the Qwen3-VL item in #794). Serves image input on
qwen3vl(dense) andqwen3vlmoeGGUFs from theirqwen3vl_mergermmproj on the #803 base: the one projector with DeepStack merger tensors. The Qwen3.5 / 3.6 / 3.8 / Flash-Next towers sharing this code still refuse a DeepStack mmproj.What it does
gguf_mm_configderivesdeepstack_visual_indexesfrom thev.deepstack.{block}.*tensors; mergerk= tap rank, as the converter writes it (deepstack_merger_list.{k}->v.deepstack.{deepstack_visual_indexes[k]}). Checked against the mmproj'sis_deepstack_layersKV and the text GGUF's{arch}.n_deepstack_layers; a tap past the tower depth or an incomplete merger refuses.rope.dimension_sections(every published family config is [24, 20, 20] interleaved). Without one, the feat(gguf): serve qwen3vl and qwen3vlmoe gguf checkpoints text only #807 text-only path is unchanged.iter_mmproj_weightsmaps the mergers tovisual.deepstack_merger_list.{k}.*and stacks the patch embedding's two temporal slices; the rest is HF layout. Tower runs bf16 via--mmprojor folder discovery.mm_processor/encoders.Tower parity
model.visual.*range-fetched from the HF safetensors: unsloth mmproj-BF16 is bit-exact against Qwen3-VL-4B-Thinking (315/315) and Qwen3-VL-30B-A3B-Thinking (351/351), the 18 DeepStack merger tensors per model included. Taps [5, 11, 17] / [8, 16, 24], matching the KV andn_deepstack_layers = 3.Check sweep
fetch_headers --search Qwen3-VL(17.7k shards) plus a header stub of every*mmproj*.gguf(100 files; the tool skips them, sofetch()was driven directly):645 trunks of
qwen3vl/qwen3vlmoe, 596 pass. No subfolder-layout repos in the corpus, so every published mmproj is folder-discoverable.Serving
RTX 5090 32 GB, driver 580.95.05, Ryzen 9 7950X3D, 187 GiB RAM.
/v1/statsreports["text", "image"].#cached-token0 -> 222); 13.4k-token prefill ~11.1k tok/s, decode 212-263 tok/s.Not covered
n_deepstack_layers = 0agrees with a tap-less tower)No per-family GGUF test file, per the #721 review direction;
tests/models/test_qwen3_vl.py's live-registry encoder test covers the rows.