Skip to content

Add Fused Decode Transformer Layer challenge (hard #160) - #385

Open
kunal-mansukhani wants to merge 2 commits into
mainfrom
add-challenge-160-fused-decode-transformer-layer
Open

kunal-mansukhani wants to merge 2 commits into
mainfrom
add-challenge-160-fused-decode-transformer-layer

Conversation

@kunal-mansukhani

Copy link
Copy Markdown
Contributor

New hard challenge #160 Fused Decode Transformer Layer: one Llama-style pre-norm decoder step for a batch of single tokens, intended as a megakernel (one persistent launch, not a chain of RMSNorm / QKV / attention / MLP kernels). The kernel RMS-normalizes x, projects QKV, applies RoPE to Q and K, writes the new key and value into the KV cache at positions[b], attends over the valid prefix 0..positions[b] with grouped-query attention, applies the output projection and residual, then a pre-norm SwiGLU MLP and residual.

It is distinct from #93 (full-sequence prefill block, no KV cache), #74 (GPT-2 prefill), #115 (QKV + RoPE + cache update only, no attention or MLP), #84 / #146 (standalone SwiGLU, no decoder or cache), and #86 / #141 / #96 (attention only).

Behavior

  • d_model = H_q × D, H_q divisible by H_kv. Weights are packed into one buffer (row-major (in, out), no biases) and applied as z × W.
  • RoPE uses the same cos_sin_cache layout as [feature request] submit challenges through CLI #115 (cos then sin in each row). It is applied to Q and K only.
  • Every cache slot other than positions[b] is left unchanged. The new K/V are written before attention, so the current token is included.
  • RMSNorm uses ε = 1e-5 and has no additive bias.
  • The statement frames a single fused launch as the intended approach; the judge still checks numerical correctness, not launch count.

Tests

  • 11 functional cases, including the hand-computed example (90-degree RoPE at position 1, GQA, residual + SwiGLU) checked against an independent loop-based float64 implementation.
  • Targeted cases: first-token decode (positions = 0); variable cache lengths in one batch; zero input at position 0 (output must be 0, other cache slots preserved); zero query over a non-empty cache (uniform attention over 0..pos); GQA vs MHA; non-power-of-two head dim / FFN / mixed positions; and two realistic Llama-style decode batches.
  • Perf case: B = 32, d_model = 2,048, H_q = 16, H_kv = 4, D = 128, d_ffn = 5,632, S_max = 2,048 (full-cache positions). Float32 working set ≈ 450 MB, about 2.25 GB at 5×, inside a 16 GB T4. No bf16 or WMMA.

Tolerance

  • atol = rtol = 1e-4. Against the float64 loop implementation, the float32 reference stays well under 1e-5 on the example and mid-size cases. That budget still rejects a skipped residual, a wrong cache length, dropped GQA repeats, or RoPE applied to V.

Verified locally

  • Independent loop-based float64 reference matches the vectorized reference_impl on the example and several functional cases; cache slots other than pos are preserved.
  • black, isort, flake8, clang-format and validate_challenges.py pass.
  • Not T4-verified yet: that needs an @claude validation comment from a maintainer.

Single-token Llama-style decoder megakernel: RMSNorm, QKV, RoPE, KV-cache
attention, residual, pre-norm SwiGLU MLP, and residual in one fused launch.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Review bot: needs a human reviewer.

  • 401 counted changed lines exceeds limit of 400

@github-actions github-actions Bot added the needs-human-review Review bot: needs a human reviewer label Nov 11, 2026
@kunal-mansukhani

Copy link
Copy Markdown
Contributor Author

@claude Please validate this new challenge on T4. Follow CLAUDE.md: verify every checklist item, write an independent CUDA solution in solution/solution.cu (do not commit it), and run python scripts/run_challenge.py challenges/hard/160_fused_decode_transformer_layer --language cuda --action run then --action submit. Report whether the example, functional tests, and performance test pass.

@shxjames shxjames left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at add2ca73e4f5c09e8a528dd752df513a1bdf2902 against an independent fp64 CPU reimplementation, using the prod grader rule (|out-ref| <= atol + rtol*max|ref|, max taken per tensor per case, checked on output, K_cache, and V_cache). Looks correct; no blockers.

Correctness

  • The example matches the HTML exactly: fp64 gives [5.8994764, 0.0558132, 0.99998, 0.5] and K/V slot 1 = 1.99996.
  • I checked the semantics directly: RMSNorm with eps inside the sqrt, half-split RoPE at positions[b] on Q and K only, the new K/V written before attention and included (s <= pos), GQA mapping h // (H_q/H_kv), 1/sqrt(D) scale, both residuals, and SiLU(gate) * up.
  • The packed-weight offset table matches unpack_weights (104 floats for the example).
  • All test dicts pass output, and all 6 starters match the signature (JAX returns a 3-tuple, same as #115).

Tolerance headroom (worst ratio, 1.0 = fail)

variant functional (11) perf
fp64 0.0033 0.0046
fp32, split-K order 0.0040 0.0049
fp16-input GEMMs 2.44 3.56
TF32 (rounded) 2.42 3.64
bf16 18.4 30.8

Correct fp32 kernels have about 200x room. fp16, TF32 and bf16 shortcuts all fail, including on perf, so this does not have #382's fp16 gap.

Bug injection: functional tests catch 29/29
Eps outside the sqrt, LayerNorm, missing norm weights, swapped gate/up, swapped W_k/W_v, transposed W_o, one shared position for the batch, RoPE position ±1, interleaved RoPE, RoPE sign flip, RoPE missing on Q or K, RoPE applied to V, writing at pos+1, attending before the write, never writing the cache, caching un-roped K, GQA h % H_kv, all heads on KV head 0, excluding the new token, no mask, 1/D scale, no scale, dropping either residual, RMSNorm 2 on x, GELU, and ReLU.

Perf

  • About 450 MB of fp32 (180 MB weights plus 2 × 134 MB caches), so ~2.3 GB at 5x on a 16 GB T4.
  • About 3.4 GFLOP and roughly 450 MB read per call: memory-bound, about 1.4 ms at minimum on a T4, not degenerate.
  • Re-running perf is idempotent: it writes the same slot with the same value.

Nits (optional)

  1. Every perf sequence uses positions = S_max-1, so on perf alone, no-mask and position/length bugs are invisible (functional still catches them). Mixed positions (e.g. uniform in [S_max/2, S_max)) would make load-balancing ragged cache lengths part of the megakernel problem and closer to real decode.
  2. HTML: "Position i of sequence b may attend…" — there's no i. Suggest "The new token of sequence b…". The GQA line could also state the mapping kv = h // (H_q/H_kv) explicitly, since "repeat" can be read as tile or interleave.
  3. Test 4's comment says "zero query, non-empty cache", but zero_x also zeroes the new K/V. The test is still valid; the comment could just mention it.

Not T4-validated by me (CPU only). CI is green.

@kunal-mansukhani

Copy link
Copy Markdown
Contributor Author

@claude Please validate this new challenge on T4. Follow CLAUDE.md: verify every checklist item, write an independent CUDA solution in solution/solution.cu (do not commit it), and run python scripts/run_challenge.py challenges/hard/160_fused_decode_transformer_layer --language cuda --action run then --action submit. Report whether the example, functional tests, and performance test pass, and post that pass/fail result as a comment on this PR.

Replace the leftover "Position i" phrasing with "The new token of
sequence b", and state the GQA mapping kv = h // (H_q / H_kv) in the
problem statement and requirements so "repeat" is not read as interleave.
@github-actions
github-actions Bot requested a review from shxjames November 12, 2026 17:20

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Review bot: needs a human reviewer.

  • 401 counted changed lines exceeds limit of 400

@kunal-mansukhani

Copy link
Copy Markdown
Contributor Author

@claude Please validate this new challenge on T4. Follow CLAUDE.md: verify every checklist item, write an independent CUDA solution in solution/solution.cu (do not commit it), and run python scripts/run_challenge.py challenges/hard/161_persistent_multi_layer_mlp --language cuda --action run then --action submit. Report whether the example, functional tests, and performance test pass.

Re-run on head b19dafd9bc76dfb1335f18b0deab86b259e1ba48 after the latest review fixes. Please post the T4 result (pass/fail with timings) as a PR comment when done.

@claude

claude Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

T4 validation for head b19dafd: ✅ PASS

Note: the latest request pointed at challenges/hard/161_persistent_multi_layer_mlp. That directory doesn't exist on this branch, so I validated challenges/hard/160_fused_decode_transformer_layer, which is what this PR adds.

Independent CUDA solution (not committed). It runs as separate kernels: RMSNorm, then a tiled fp32 GEMM for Q/K/V, then RoPE plus the K/V cache write at positions[b], then per-(b, h) GQA attention over 0..pos, then W_o with the residual, RMSNorm 2, gate/up GEMMs, SiLU⊙up, and W_down with the residual.

Step Result
--action run (example) ✓ Test passed (the first attempt timed out after the platform retried on a fresh host; the rerun passed with no code change)
--action submit (11 functional + performance) ✓ All tests passed

The platform didn't return timings for this submission.

Checklist (CLAUDE.md)

  • HTML: starts with <p>; has <h2> sections; SVG uses a #222 background; the first example matches generate_example_test() (I recomputed output, K_cache and V_cache by hand); the perf bullet matches generate_performance_test(); \text{cos_sin_cache} uses plain underscores; the packed-weight offsets in the example (104 floats) are correct.
  • challenge.py: class attributes only; asserts cover shape and dtype only; all allocations use self.device; standard PyTorch ops only; 11 functional cases with comments explaining what each catches; hand-computed expected values are checked against the reference; the perf working set is about 450 MB (fits 5× in 16 GB).
  • Starters: all 6 present with exactly one parameter comment each ("device pointers" without the parenthetical, as required for hard). The JAX starter follows the same 3-tuple return convention as [feature request] submit challenges through CLI #115.
  • Lint: black, isort, flake8 and clang-format pass. The mojo format hook couldn't run here because mojo isn't installed in this environment.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-human-review Review bot: needs a human reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants