ShiftAddQwen

Models: huggingface.co/EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF · Code: this repository

ShiftAddQwen applies the ShiftAddLLM reparameterization to Qwen3.5-9B (hybrid Gated-DeltaNet / full attention, with MTP head) and deploys it through llama.cpp. Every decoder weight group is a sum of binary planes with per-group scales, w ≈ Σ_p α_p · b_p, b_p ∈ {−1, +1}, so a mat-vec reduces to sign-select-and-add per plane plus one scale per group. We contribute (i) an output-aware fit of this form for a hybrid-attention model, (ii) KLD-measured mixed precision and end-to-end scale distillation, and (iii) new ggml quant types with CUDA kernels, so the models run unmodified in llama.cpp (patched).

Naming. Our models are ShiftAddQwen-3.5-9B-xbits, x being the average bits per decoder weight. Baselines are UnslothAI/Qwen-3.5-9B-recipe: the public GGUF files of the same checkpoint from unsloth/Qwen3.5-9B-MTP-GGUF, i.e. the standard llama.cpp recipes (Q8_0, Q4_K_M, Q3_K_M) and Unsloth's imatrix-calibrated dynamic recipes (UD-Q3_K_XL, UD-IQ3_XXS, UD-Q2_K_XL, UD-IQ2_M). Q4_K_M (5.87 GB) is the reference 4-bit deployment.

News

• 2026-10-23. The sa* types are rebased on current llama.cpp master (36e36e63a) as two patches: llama.cpp/shiftadd-master-cpu.patch adds the types with CPU kernels (generic, AVX2, AVX2 + VNNI, AVX-512 + VNNI, NEON), and llama.cpp/shiftadd-master-cuda.patch, applied on top, adds the CUDA kernels (MMVQ decode, MMQ prefill for batches below 64, dequantize + cuBLAS above, and MoE MUL_MAT_ID). Both are tested against the CPU backend on V100 and A100 and pass llama.cpp's CPU CI. Upstreaming is proposed in llama.cpp discussion #31051. The shiftadd-b11538.patch build used for the results below is unchanged.
• 2026-10-22. Tool-calling results: BFCL v4 and τ-bench for the 4.4- and 4.0-bit models against Q4_K_M and Q3_K_M (docs/results.md), and a note on the sign-select mat-vec and how ShiftAddQwen relates to ShiftAddLLM (docs/shiftadd_matvec.pdf).
• 2026-10-20. First release: ShiftAddQwen-3.5-9B at 3.0, 3.2, 3.6, 4.0 and 4.4 bits on the Hub, with the quantization pipeline and the patched llama.cpp (b11538) in this repository.

Results

Figure 1. ShiftAddQwen-3.5-9B-xbits (x = average bits per decoder weight) against the UnslothAI/Qwen-3.5-9B- GGUF quants. (a) KL divergence to the BF16 model on held-out mixed-domain text; (b) mean accuracy on GSM8K, MMLU-Pro, HumanEval and MGSM, with bootstrap 95% confidence intervals over the 844 items; (c) decode and (d) prefill throughput (A100, CUDA; the batched path dequantizes to fp16 and uses cuBLAS). Vector: docs/frontier.pdf.*

Full measurements for every model in Figure 1 (KLD, PPL, per-task accuracy, prefill/decode/MTP throughput) are in docs/results.md; raw numbers in results/summary.json. Tool calling (BFCL v4, τ-bench) of the 4.4- and 4.0-bit models against Q4_K_M and Q3_K_M: docs/results.md#agentic-benchmarks-tool-calling (tools/agentic).

Findings.

1. Memory. ShiftAddQwen-3.5-9B-4.0bits is 19% smaller than UnslothAI/Qwen-3.5-9B-Q4_K_M (4.76 vs 5.87 GB) at near-equal quality (KLD 0.038 vs 0.026; tasks within noise), and the 4.4-bit model is 13% smaller at KLD 0.030 with tasks within noise of Q4_K_M. ShiftAddQwen-3.5-9B-3.6bits is 25% smaller than Q4_K_M and matches UnslothAI/Qwen-3.5-9B-Q3_K_M in task accuracy at lower KLD (0.056 vs 0.067).
2. Fidelity frontier. At every size from 3.9 to 5.1 GB, ShiftAddQwen has lower KLD than the UnslothAI quant of equal or larger size (e.g. 0.086 at 4.07 GB vs UD-IQ3_XXS 0.088 at 4.30 GB; 0.030 at 5.10 GB vs UD-Q3_K_XL 0.036 at 5.24 GB).
3. Speed. The kernels decode 4–20% faster than Q4_K_M and prefill 26–31% faster (5.3–5.5K vs 4.2K tok/s), and MTP speculative decoding keeps its acceptance rate (0.60–0.68).

Protocol (details in docs/results.md). KLD/PPL: llama-perplexity against the BF16 GGUF, 512-token windows × 40, on wikitext-2 and on held-out mixed-domain text. Tasks: GSM8K, MMLU-Pro, HumanEval, MGSM (844 items), greedy, thinking off; one run per model, so differences below ~1.5 points are noise. Speed: A100-PCIE-40GB, CUDA, llama-bench and llama-server (65K context, with and without MTP drafting).

Method

ShiftAddQwen keeps ShiftAddLLM's weight representation (binary planes with per-group scales) and replaces its fitting pipeline and kernels: the fit is output-aware with measured mixed precision and end-to-end scale distillation, and the decode kernel is LUT-free (sign-select with dp4a byte masks). The symbols, the mat-vec identity, and what is kept from / replaced in ShiftAddLLM are written up in docs/shiftadd_matvec.pdf (source: docs/shiftadd_matvec.tex).

Format. ggml types sa{2,3,4,5,6}[_128|_256] (2.1–6.8 bits/weight): a super-block of 256 weights along the input dimension stores Q sign planes (Q·32 bytes), one u8 scale code per plane and group of G ∈ {64, 128, 256} weights, and one fp16 scale d; α = d·a. The CUDA mat-vec turns each nibble of sign bits into a byte mask and accumulates with dp4a against q8 activations, applying the scale once per group and plane; prefill dequantizes to fp16 and uses cuBLAS. A generalized block-wise family sab<Q>_<bc>x<br> (scales shared by bc input columns × br output rows, covering ShiftAddLLM's column-wise and block-wise designs) is provided as well.

Output-aware fit (saq/gptq_bcq.py, saq/colwise.py). A GPTQ column sweep with act-order over groups; at each group boundary the BCQ scales are fitted on the error-compensated weights (greedy initialisation, alternating least squares with exhaustive code choice), and after the sweep all scales are re-solved exactly for the Hessian-weighted output error with the codes frozen. Layers are quantized sequentially on the outputs of the already-quantized prefix.

Mixed precision (saq/sens_kld.py, saq/sensitivity.py). The end-to-end KLD increase of quantizing each linear alone at every (bits, group) setting is measured, the bf16 forward-noise floor (≈3·10⁻⁴) subtracted, and a Lagrangian knapsack selects the setting per linear for a target average bits/weight.

Scale distillation (saq/finetune.py, saq/qlinear.py). With the binary codes frozen, every scale is trained as α₀(1+θ) to match the BF16 teacher's top-64 log-probabilities plus a rest-of-vocabulary bucket, on 768 × 2048 tokens, then snapped to the u8 grid.

Calibration. Web/wiki 25%, English chat 20%, code 15%, multilingual chat (Aya, 20 languages) 25%, math (MetaMathQA) 15%, so that the Hessians and the distillation targets cover the domains the model serves.

High precision kept for the DeltaNet gates, conv1d and norms; the embedding (Q3_K) and LM head (Q5_K) use llama.cpp k-quants with the Unsloth imatrix; the MTP block uses the Q4_K_M recipe.

Run a released model

Released GGUFs (Hub: EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF): ShiftAddQwen-3.5-9B-{3.0,3.2,3.6,4.0,4.4}bits.gguf (3.90–5.10 GB; see Table 1 for the trade-offs). The models use new ggml quant types, so they need the patched llama.cpp from this repository (stock llama.cpp refuses to load them). 6–8 GB of GPU memory is enough for a 32K context.

scripts/setup_llama.sh                   # llama.cpp b11538 + patch, CUDA build (CUDA_ARCH=80 default; set yours)
hf download EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF ShiftAddQwen-3.5-9B-4.4bits.gguf --local-dir models
work/llama.cpp/build/bin/llama-server -m models/ShiftAddQwen-3.5-9B-4.4bits.gguf -ngl 99 -fa on -c 32768 \
    --jinja --spec-type draft-mtp --spec-draft-n-max 3      # MTP speculative decoding, as in Table 1

To use current llama.cpp master instead, apply llama.cpp/shiftadd-master-cpu.patch and then llama.cpp/shiftadd-master-cuda.patch to a checkout of commit 36e36e63a and build as usual.

Reproduce

Running or evaluating the released GGUF models needs 6–8 GB of GPU memory (any CUDA GPU; see Run a released model). Reproducing the quantization pipeline as released needs one CUDA GPU with ≥40 GB (the bf16 model stays resident during quantization, sensitivity measurement and distillation), ~80 GB of disk, Python 3.12, CUDA toolkit and CMake.

pip install -r requirements.txt          # pinned versions; see the file for the torch/CUDA wheel
source scripts/env.sh                    # paths default to ./work; override SAQ_WORK, SAQ_MODEL, ...
scripts/download.sh                      # Qwen/Qwen3.5-9B + unsloth/Qwen3.5-9B-MTP-GGUF baselines
scripts/setup_llama.sh                   # llama.cpp b11538 + llama.cpp/shiftadd-b11538.patch, CUDA build, kernel tests
python tools/make_eval_text.py           # wiki.test.raw, mix.test.raw
scripts/pipeline.sh 4.0                  # sensitivity -> plan -> quantize -> distill -> GGUF -> KLD / tasks / speed
scripts/eval_baselines.sh                # same measurements for the Unsloth quants
python tools/collect.py && python tools/plot.py

FT_NSEQ=64 scripts/pipeline.sh 4.0 runs a short end-to-end check. The sensitivity step runs once and is shared by all target sizes; its four shards (--layers 0-7, …) can run on separate GPUs. slurm/example.sbatch wraps any step for a cluster.

Layout

path	contents
saq/bcq.py, saq/gptq_bcq.py	BCQ primitives; output-aware fit of one linear (row-wise scales)
saq/colwise.py	block-wise / column-wise scales: fit, exact refit, packing
saq/quantize_model.py	sequential quantization of all decoder linears
saq/sens_kld.py, saq/sensitivity.py	KLD sensitivity, knapsack planner
saq/qlinear.py, saq/finetune.py	trainable ShiftAdd linears, scale distillation
saq/export_gguf.py	GGUF writer for the new types (bit-exact packing and layout checks)
saq/data.py, saq/model.py	calibration mixes, Qwen3.5 helpers, PyTorch KLD
llama.cpp/shiftadd-b11538.patch	ggml types: CPU reference, CUDA dequant / get_rows / mat-vec, gguf-py, tests
llama.cpp/shiftadd-master-{cpu,cuda}.patch	the same types on llama.cpp master: CPU kernels (AVX2 / AVX-512 / NEON), then CUDA (MMVQ, MMQ, MoE); apply in that order
scripts/, tools/	setup, pipeline, evaluation, benchmarks, plots

Citation

ShiftAddQwen builds on ShiftAddLLM but is not the same method (see docs/shiftadd_matvec.pdf for what is kept and what is replaced); please cite both.

@misc{you2026shiftaddqwen,
  title={ShiftAddQwen: ShiftAdd reparameterization of Qwen3.5-9B for llama.cpp},
  author={You, Haoran},
  year={2026},
  howpublished={\url{https://github.com/Purdue-EcoAI/ShiftAddQwen}},
  note={Models: \url{https://huggingface.co/EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF}}
}

@inproceedings{you2024shiftaddllm,
  title={ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization},
  author={You, Haoran and Guo, Yipin and Fu, Yichao and Zhou, Wei and Shi, Huihong and Zhang, Xiaofan and Kundu, Souvik and Yazdanbakhsh, Amir and Lin, Yingyan Celine},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2024}
}