Repository navigation
Add Fused Decode Transformer Layer challenge (hard #160) - #385
kunal-mansukhani wants to merge 2 commits into
Conversation
Single-token Llama-style decoder megakernel: RMSNorm, QKV, RoPE, KV-cache attention, residual, pre-norm SwiGLU MLP, and residual in one fused launch.
|
@claude Please validate this new challenge on T4. Follow CLAUDE.md: verify every checklist item, write an independent CUDA solution in |
shxjames
left a comment
There was a problem hiding this comment.
Reviewed at add2ca73e4f5c09e8a528dd752df513a1bdf2902 against an independent fp64 CPU reimplementation, using the prod grader rule (|out-ref| <= atol + rtol*max|ref|, max taken per tensor per case, checked on output, K_cache, and V_cache). Looks correct; no blockers.
Correctness
- The example matches the HTML exactly: fp64 gives
[5.8994764, 0.0558132, 0.99998, 0.5]and K/V slot 1 =1.99996. - I checked the semantics directly: RMSNorm with eps inside the sqrt, half-split RoPE at
positions[b]on Q and K only, the new K/V written before attention and included (s <= pos), GQA mappingh // (H_q/H_kv), 1/sqrt(D) scale, both residuals, and SiLU(gate) * up. - The packed-weight offset table matches
unpack_weights(104 floats for the example). - All test dicts pass
output, and all 6 starters match the signature (JAX returns a 3-tuple, same as #115).
Tolerance headroom (worst ratio, 1.0 = fail)
| variant | functional (11) | perf |
|---|---|---|
| fp64 | 0.0033 | 0.0046 |
| fp32, split-K order | 0.0040 | 0.0049 |
| fp16-input GEMMs | 2.44 | 3.56 |
| TF32 (rounded) | 2.42 | 3.64 |
| bf16 | 18.4 | 30.8 |
Correct fp32 kernels have about 200x room. fp16, TF32 and bf16 shortcuts all fail, including on perf, so this does not have #382's fp16 gap.
Bug injection: functional tests catch 29/29
Eps outside the sqrt, LayerNorm, missing norm weights, swapped gate/up, swapped W_k/W_v, transposed W_o, one shared position for the batch, RoPE position ±1, interleaved RoPE, RoPE sign flip, RoPE missing on Q or K, RoPE applied to V, writing at pos+1, attending before the write, never writing the cache, caching un-roped K, GQA h % H_kv, all heads on KV head 0, excluding the new token, no mask, 1/D scale, no scale, dropping either residual, RMSNorm 2 on x, GELU, and ReLU.
Perf
- About 450 MB of fp32 (180 MB weights plus 2 × 134 MB caches), so ~2.3 GB at 5x on a 16 GB T4.
- About 3.4 GFLOP and roughly 450 MB read per call: memory-bound, about 1.4 ms at minimum on a T4, not degenerate.
- Re-running perf is idempotent: it writes the same slot with the same value.
Nits (optional)
- Every perf sequence uses
positions = S_max-1, so on perf alone, no-mask and position/length bugs are invisible (functional still catches them). Mixed positions (e.g. uniform in[S_max/2, S_max)) would make load-balancing ragged cache lengths part of the megakernel problem and closer to real decode. - HTML: "Position
iof sequencebmay attend…" — there's noi. Suggest "The new token of sequenceb…". The GQA line could also state the mappingkv = h // (H_q/H_kv)explicitly, since "repeat" can be read as tile or interleave. - Test 4's comment says "zero query, non-empty cache", but
zero_xalso zeroes the new K/V. The test is still valid; the comment could just mention it.
Not T4-validated by me (CPU only). CI is green.
|
@claude Please validate this new challenge on T4. Follow CLAUDE.md: verify every checklist item, write an independent CUDA solution in |
Replace the leftover "Position i" phrasing with "The new token of sequence b", and state the GQA mapping kv = h // (H_q / H_kv) in the problem statement and requirements so "repeat" is not read as interleave.
|
@claude Please validate this new challenge on T4. Follow CLAUDE.md: verify every checklist item, write an independent CUDA solution in Re-run on head |
T4 validation for head
|
| Step | Result |
|---|---|
--action run (example) |
✓ Test passed (the first attempt timed out after the platform retried on a fresh host; the rerun passed with no code change) |
--action submit (11 functional + performance) |
✓ All tests passed |
The platform didn't return timings for this submission.
Checklist (CLAUDE.md)
- HTML: starts with
<p>; has<h2>sections; SVG uses a#222background; the first example matchesgenerate_example_test()(I recomputed output, K_cache and V_cache by hand); the perf bullet matchesgenerate_performance_test();\text{cos_sin_cache}uses plain underscores; the packed-weight offsets in the example (104 floats) are correct. - challenge.py: class attributes only; asserts cover shape and dtype only; all allocations use
self.device; standard PyTorch ops only; 11 functional cases with comments explaining what each catches; hand-computed expected values are checked against the reference; the perf working set is about 450 MB (fits 5× in 16 GB). - Starters: all 6 present with exactly one parameter comment each ("device pointers" without the parenthetical, as required for hard). The JAX starter follows the same 3-tuple return convention as [feature request] submit challenges through CLI #115.
- Lint: black, isort, flake8 and clang-format pass. The
mojo formathook couldn't run here becausemojoisn't installed in this environment.
🤖 Generated with Claude Code
New hard challenge #160 Fused Decode Transformer Layer: one Llama-style pre-norm decoder step for a batch of single tokens, intended as a megakernel (one persistent launch, not a chain of RMSNorm / QKV / attention / MLP kernels). The kernel RMS-normalizes
x, projects QKV, applies RoPE to Q and K, writes the new key and value into the KV cache atpositions[b], attends over the valid prefix0..positions[b]with grouped-query attention, applies the output projection and residual, then a pre-norm SwiGLU MLP and residual.It is distinct from #93 (full-sequence prefill block, no KV cache), #74 (GPT-2 prefill), #115 (QKV + RoPE + cache update only, no attention or MLP), #84 / #146 (standalone SwiGLU, no decoder or cache), and #86 / #141 / #96 (attention only).
Behavior
d_model = H_q × D,H_qdivisible byH_kv. Weights are packed into one buffer (row-major(in, out), no biases) and applied asz × W.cos_sin_cachelayout as [feature request] submit challenges through CLI #115 (cos then sin in each row). It is applied to Q and K only.positions[b]is left unchanged. The new K/V are written before attention, so the current token is included.ε = 1e-5and has no additive bias.Tests
positions = 0); variable cache lengths in one batch; zero input at position 0 (output must be 0, other cache slots preserved); zero query over a non-empty cache (uniform attention over0..pos); GQA vs MHA; non-power-of-two head dim / FFN / mixed positions; and two realistic Llama-style decode batches.B= 32,d_model= 2,048,H_q= 16,H_kv= 4,D= 128,d_ffn= 5,632,S_max= 2,048 (full-cache positions). Float32 working set ≈ 450 MB, about 2.25 GB at 5×, inside a 16 GB T4. No bf16 or WMMA.Tolerance
atol = rtol = 1e-4. Against the float64 loop implementation, the float32 reference stays well under 1e-5 on the example and mid-size cases. That budget still rejects a skipped residual, a wrong cache length, dropped GQA repeats, or RoPE applied to V.Verified locally
reference_implon the example and several functional cases; cache slots other thanposare preserved.validate_challenges.pypass.@claudevalidation comment from a maintainer.