[gdevenyi]

Stacked on #82. The base 5080/base-ecb52158 is the previous 5080deploy tip. This PR rebuilds 5080deploy on upstream main 07813248, which has 4 new commits:

• perf(kernel): avoid layout conversions in nvfp4 marlin decode FlashML-org/FreeToken#631: NVFP4 marlin decode
• perf(kernel): read dsv41 indexer key rows as int32 words FlashML-org/FreeToken#634: dsv41 indexer int32 rows
• perf(kernel): keep 16-byte kv loads in dsv4 sparse attention and unroll gated pool FlashML-org/FreeToken#633: dsv4 sparse attention loads
• perf(kernel): skip masks and out-of-window tiles in triton extend attention FlashML-org/FreeToken#632: triton extend attention

Superseded work: none this round. None of the 4 commits is patch-equivalent to a fork commit, and all 103 replayed steps still change something on the new base. The carried PRs whose upstream heads moved (FlashML-org#408, FlashML-org#435, FlashML-org#572, FlashML-org#578) have identical content, so their recorded heads are kept.

Replay: the same method as #81. Two hand resolutions:

• kernel/triton/attention.py (at the feat(kvcache): add nvfp4 kv quantization FlashML-org/FreeToken#408 step):
  • perf(kernel): skip masks and out-of-window tiles in triton extend attention FlashML-org/FreeToken#632 restructured the split extend kernel into _extend_tile_ranges and _extend_attend_tile.
  • Git silently auto-merged its quant-unaware unmasked prefix loop, outside the conflict hunk. Taking upstream's side compiles, but it reads raw fp8/NVFP4 codes as bf16.
  • Resolution: a new _extend_attend_cache_tile (the fp8/NVFP4 twin; fp8 scales on the dot outputs, l_i from unscaled p), dispatched on HAS_KV_SCALE in both cache loops. The new-token loops stay as upstream wrote them.
  • GPU check against a dequantized-cache reference: 36/36 cases pass (16-bit, fp8 and NVFP4 caches; head dim 64 and 128; prefixes 37, 200 and 1000, which reach the unmasked loop; window 0 and 100). This kernel is not on qwen4_exp's path (qsa_sparse).
• kernel/triton/nvfp4_fused_moe.py (at the perf(moe): dequant nvfp4 prefill experts by bit placement and load activations contiguously FlashML-org/FreeToken#578 step): union of the two import lines.

Effect in production. Only FlashML-org#631 changes what this deployment runs: it is the Triton NVFP4 MoE decode GEMV, used in every layer at every decode step. An r4-without-FlashML-org/FreeToken#631 arm reproduces the previous greedy fingerprints bit-for-bit in both rounds.

Interleaved A/B, 2 rounds, RTX 5080, production config (2100 slots, rule eviction, prefetch, host embedding):

	prod ecb52158	this PR
decode 128K (tok/s)	51.15-51.50	52.61-52.98 (+2.8%)
decode 64K	51.86 / 51.98	53.39 / 53.44 (+2.9%)
decode 32K (rep 1 / rep 2)	53.4 / 53.7	52.4 / 55.3
decode 255K + side calls	47.22	48.79 (+3.3%)
255K peak VRAM	15616 MiB	15616 MiB
quality.py	14/15	14/15
misses/token	93.8	95.1

• Greedy outputs change, as expected from the restructured K reduction: 47 of 60 cold nonce prompts differ.
• New cold fingerprints: code ae6eb52fe5986364, ctx 65dfde98bc9ee057, think c0b1deb72c69c83d.
• In 1 of 6 cold starts the think fingerprint differed (Cold greedy output occasionally differs between server starts under the new decode numerics (upstream #631/#585), think prompt only #83). A micro-test shows perf(kernel): avoid layout conversions in nvfp4 marlin decode FlashML-org/FreeToken#631's kernel itself is deterministic, and that its wide and narrow activation loads give the same bits.

Tests:

• Kernel tests: 179 passed (nvfp4 backends, e4m3, triton attention, kv fp8/nvfp4, qsa nvfp4, qsa kernels).
• Full not-slow GPU suite: 3506 passed. The only failures are the 4 known environment ones and upstream's test_dsfp4_reference_semantics GEMV case, which fails the same way on pure upstream on sm_120.

Not merged this round:

• Deferred: fix(kernel): correct jit tensor matcher checks and pdl arch guards FlashML-org/FreeToken#638 (no functional change on sm_120; full JIT rebuild) and build(deps): move to torch 2.14.1, triton 3.8 and sglang-kernel 0.4.9 FlashML-org/FreeToken#630 (torch/triton bump).
• Skipped: fix(rotary): preserve fp32 inputs in mrope torch fallbacks FlashML-org/FreeToken#641, perf(moe): avoid intra-op fanout while filling NVFP4 host banks FlashML-org/FreeToken#635, feat(quant): opt-in fp8-block QAT for the lm_head  FlashML-org/FreeToken#636, feat(qwen3_5_moe): tensor-parallel weight sharding for multi-GPU serving FlashML-org/FreeToken#637 and feat(server): add --api-key authentication FlashML-org/FreeToken#639 (paths this deployment does not run), plus ROCm and DeepSeek PRs.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt