Skip to content

feat(gguf): serve image input on qwen3vl ggufs from a deepstack qwen3vl_merger mmproj - #827

Merged
jason-fxz merged 1 commit into
FlashML-org:mainfrom
void-mckenzie:feat/gguf-qwen3vl-vision
Oct 26, 2026
Merged

jason-fxz merged 1 commit into
FlashML-org:mainfrom
void-mckenzie:feat/gguf-qwen3vl-vision

Conversation

@void-mckenzie

Copy link
Copy Markdown
Contributor

Part of #671 (the Qwen3-VL item in #794). Serves image input on qwen3vl (dense) and qwen3vlmoe GGUFs from their qwen3vl_merger mmproj on the #803 base: the one projector with DeepStack merger tensors. The Qwen3.5 / 3.6 / 3.8 / Flash-Next towers sharing this code still refuse a DeepStack mmproj.

What it does

  • gguf_mm_config derives deepstack_visual_indexes from the v.deepstack.{block}.* tensors; merger k = tap rank, as the converter writes it (deepstack_merger_list.{k} -> v.deepstack.{deepstack_visual_indexes[k]}). Checked against the mmproj's is_deepstack_layers KV and the text GGUF's {arch}.n_deepstack_layers; a tap past the tower depth or an incomplete merger refuses.
  • With a tower the text runs interleaved mrope, sections from rope.dimension_sections (every published family config is [24, 20, 20] interleaved). Without one, the feat(gguf): serve qwen3vl and qwen3vlmoe gguf checkpoints text only #807 text-only path is unchanged.
  • iter_mmproj_weights maps the mergers to visual.deepstack_merger_list.{k}.* and stacks the patch embedding's two temporal slices; the rest is HF layout. Tower runs bf16 via --mmproj or folder discovery.
  • Both GGUF registry rows gain mm_processor / encoders.

Tower parity

model.visual.* range-fetched from the HF safetensors: unsloth mmproj-BF16 is bit-exact against Qwen3-VL-4B-Thinking (315/315) and Qwen3-VL-30B-A3B-Thinking (351/351), the 18 DeepStack merger tensors per model included. Taps [5, 11, 17] / [8, 16, 24], matching the KV and n_deepstack_layers = 3.

Check sweep

fetch_headers --search Qwen3-VL (17.7k shards) plus a header stub of every *mmproj*.gguf (100 files; the tool skips them, so fetch() was driven directly):

result files
pass, vision exercised (41 repos) 477
pass, repo ships no mmproj 119
IQ1_M kernel gap 29
broken uploads (fail identically on main) 12
ik_llama custom types 10

645 trunks of qwen3vl / qwen3vlmoe, 596 pass. No subfolder-layout repos in the corpus, so every published mmproj is folder-discoverable.

Serving

RTX 5090 32 GB, driver 580.95.05, Ryzen 9 7950X3D, 187 GiB RAM.

  • 4B UD-Q4_K_XL + auto-discovered mmproj-BF16: reads OCR text verbatim, /v1/stats reports ["text", "image"].
  • 30B-A3B UD-Q3_K_XL (MoE offload): two images answered in order; a repeated image prompt hits the radix through the span (#cached-token 0 -> 222); 13.4k-token prefill ~11.1k tok/s, decode 212-263 tok/s.

Not covered

  • video (the rows register the vision encoder only, as the HF checkpoints)
  • Qwen3-ASR / Omni audio towers; their text files parse (n_deepstack_layers = 0 agrees with a tap-less tower)
  • the 80 repos that ship no mmproj

No per-family GGUF test file, per the #721 review direction; tests/models/test_qwen3_vl.py's live-registry encoder test covers the rows.

@void-mckenzie
void-mckenzie marked this pull request as ready for review October 26, 2026 20:11
@jason-fxz

Copy link
Copy Markdown
Collaborator

Cross-checked at 54bb59f on an RTX PRO 6000 Blackwell (96 GB) against a local implementation of the same change:

  • Tower: unsloth BF16 / F32 mmproj vs Qwen/Qwen3-VL-8B-Instruct and Qwen/Qwen3-VL-30B-A3B-Instruct, 351/351 bit-exact each; F16 max 3e-8, Qwen's own Q8_0 max 2.4e-3. vision_config, image processor (12 images) and image token ids equal HF.
  • Qwen3.5 / 3.6 / 3.8: derived configs, refusals, tower weights and auto-discovery byte-identical to main on 13 pairings.
  • Header stubs of 120 Qwen3-VL repos: in all 259 qwen3vl_merger mmproj files is_deepstack_layers matches the v.deepstack.* tensors and every merger is complete; every qwen3vl / qwen3vlmoe text file declares n_deepstack_layers = 3. gguf_check over the 380 text x mmproj pairs gives the same result per file as our version.
  • e2e, 10 setups x 17 probes (8B unsloth Q8_0 and Qwen Q4_K_M, 30B-A3B Q4_K_M; auto-discovered BF16, F16 / Q8_0 via --mmproj, --mm-disable vision, no mmproj): prompt tokens equal FreeToken serving the HF checkpoint 17/17, greedy answers identical to our implementation, the two text-only setups reject images with a 400.
  • tests/models tests/mm tests/tokenizer tests/server: the same 7 failures as main (test_quant_config.py cases that read the local HF cache).

LGTM.

@jason-fxz
jason-fxz merged commit 380693a into FlashML-org:main Oct 26, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants