[jason-fxz]

Part of #671 (the Muse-Glimmer item in #794). Serves the Muse-Glimmer vision tower from a llama.cpp muse-glimmer mmproj on the #803 base. Images only.

What it does

• gguf_mm_config derives the tower config from the mmproj metadata and tensor shapes (position table side, adapter widths); image_token_id is <|patch|> in the text GGUF's tokenizer. Temporal patch 2, rope theta 10000, max_image_tokens 4096, gelu and the layer pattern (window attention except every 4th block and the last) are not in the mmproj: they are family constants equal to the HF release and to what llama.cpp hardcodes in clip.cpp. MuseGlimmerImageProcessor builds from the derived preprocessor config, which equals the HF processor_config.json.
• iter_mmproj_weights undoes the converter's q / k row permute for llama.cpp's interleaved rope. The converter also sums the patch embedding's two temporal slabs into one kernel. An image repeats its frame in both slabs, so W0.x + W1.x = (W0 + W1).x: the loader writes the stored f32 sum s back as hi = bf16(s), lo = bf16(s - hi) in the two slabs, and hi + lo == s in all 903,168 weights.
• The GGUF registry row gains encoders / mm_processor and switches to the ConditionalGeneration class.
• IMAGE_START_ID / IMAGE_END_ID move from the processor to muse_glimmer/config.py, which both the processor and gguf.py import (the DeepSeek-V4 processor reads its image id the same way).
• Refusals (an error under --mmproj, a logged skip under auto-discovery): another projector, an output width other than the text model's, no <|patch|> token, <|image_start|> / <|image_end|> not at the ids the processor uses (200080 / 200081), a patch embedding that is not the summed layout; unexpected tensors fail the load.
• Images only, as on the HF path: a video_url part is a 400. The summed slabs could not serve video anyway.

Tower parity

Against the bf16 HF checkpoint:

mmproj	weights	tower output, 6 images (rel. to fp32 on HF weights)
unsloth BF16, ggml-org BF16	808/808 bit-exact besides the patch embedding	1e-6 .. 2e-5
bartowski F16	max 3.8e-6 (f16 subnormals)	2e-6 .. 3e-5
ggml-org / unsloth Q8_0	max 9.8e-3, cos >= 0.99986	1.4e-2 .. 3.9e-2
lmstudio / unsloth Q4_K	max 0.285, cos >= 0.9967	7e-2 .. 2.2e-1

Writing the BF16 sum into one slab instead (bf16(s), zero in the other) would give 1.5e-3 .. 1.5e-2. Image processor vs AutoImageProcessor: pixels and grids identical on 19 images at two budgets. Image token ids and the chat template's image rendering equal HF.

Check sweep

scripts/gguf_check on header stubs of 1278 Muse-Glimmer GGUF files in 133 repos plus their 130 mmproj files, kernels.json probed on the GPU.

run	result
main 0bf7dbb vs this PR, text side	1132/1172 muse-glimmer files pass on both, identical per file
auto-discovery, vision	577 files pick an mmproj; 576 pass (1 IQ1_M text kernel gap)
FREETOKEN_MMPROJ	24/26 subfolder-layout files pass with the tower; all 130 mmproj files pass on one text file

Serving

RTX PRO 6000 Blackwell Server Edition (96 GB), driver 595.91.07, Xeon Platinum 8559C, 249 GB RAM, torch 2.14.1+cu130, transformers 5.18.0.

ft serve --model-path <dir>/Muse-Glimmer-30B-UD-Q4_K_XL.gguf --max-seq-len-override 32768 [--mmproj <file>]

13 requests (OCR, shapes, two images, wide / tiny / tall images, a repeat, text, video), compared with FreeToken serving RedHatAI/Muse-Glimmer-30B-NVFP4:

text GGUF	mmproj	image prompt tokens = HF	answers
unsloth UD-Q4_K_XL	BF16, auto-discovered	12/12	correct
ggml-org Q4_K_M	BF16, auto-discovered	12/12	correct
unsloth UD-Q4_K_XL	bartowski F16 / ggml-org Q8_0, --mmproj	12/12	correct

--mm-disable vision and a folder without an mmproj serve text and reject images with a 400.

Not covered

item	why
video	not served from the HF checkpoint either; the mmproj also keeps only the summed patch slabs
subfolder layouts, upper-case MMPROJ-* names	discovery reads the model file's own folder with the *mmproj*.gguf glob (#803); --mmproj picks the file

No per-family GGUF test file, per the #721 review direction. pytest tests/models tests/mm tests/tokenizer tests/server: 1157 passed, 7 failed, the same 7 as on main 0bf7dbb (test_quant_config.py cases that read the local HF cache).