Repository navigation
Add Persistent Multi-Layer MLP challenge (hard #161) - #386
kunal-mansukhani wants to merge 2 commits into
Conversation
Stack of pre-norm residual SwiGLU MLP layers in one persistent megakernel, using grid-wide sync between layers instead of a launch per layer.
There was a problem hiding this comment.
✅ Review bot: low risk, safe to merge. Lint and structure checks passed; no sensitive paths or grader edits; 198 lines changed.
- ID 161 is not used by any existing challenge. It is distinct from #84, #54, #65, #146, #93 and the other listed challenges, because it stacks L pre-norm residual SwiGLU layers.
- It is non-trivial: a multi-layer fused MLP with grid-wide synchronization is a real GPU exercise.
- challenge.py has the 5 required methods, no init override, and name/atol/rtol/num_gpus/access_tier set. The extra _make_test_case helper is harmless.
- Imports are only ctypes, typing, torch, torch.nn.functional and core.challenge_base. There is no network, file, subprocess or import-time code.
- reference_impl asserts shapes and dtypes, has no device checks, and uses standard PyTorch ops only.
- All generate_* allocations use device=self.device.
- Functional tests cover L=1, zero input, power-of-two and non-power-of-two sizes, L=5, and large stacks. The hand-computed example checks out: 1 + 1.609 ≈ 2.6088.
- atol=rtol=1e-4 is reasonable for float32 and would still reject a skipped layer, a post-norm residual or a missing SiLU.
- The performance test uses about 294 MB, well under the 3.2 GB limit.
- Starters are empty stubs with only the parameter comment. The Mojo starter uses max.gpu imports and MutAnyOrigin, and challenge.html matches the signature and example.
|
@claude Please validate this new challenge on T4. Follow CLAUDE.md: verify every checklist item, write an independent CUDA solution in |
T4 validation: Persistent Multi-Layer MLP (#161)Result: passes on T4. I wrote a CUDA solution as a single persistent kernel:
This also confirms the intended cooperative-groups approach compiles and runs on the platform without Checklist (CLAUDE.md)
Optional nits (non-blocking)
|
shxjames
left a comment
There was a problem hiding this comment.
Reviewed at 4535833. Independent fp64 CPU stack (pre-norm RMSNorm, SiLU-SwiGLU, residual add). Margins are max-scaled per output (atol + rtol·max|ref|, atol = rtol = 1e-4).
Spec / reference
- Clear and matches the reference: no bias, ε = 1e-5, pre-norm residual (add uses h, not the normalized values), layers 0…L−1 in order,
z × W[l]with(in, out)layouts (W_gate/W_upare[L, d_model, d_ffn],W_downis[L, d_ffn, d_model]). Last layer uses the same residual + SiLU formula;output = h^{(L)}. - The example closed form is
1 + σ(r)·r²withr = 1/sqrt(1/2+ε)= 2.60882274; the reference is 2.60882258. - Every dict passes all 10 params, including
output. The reference leaves the inputs untouched. - Persistence: the write-up correctly says the judge checks numerics, not launch count. A per-layer launch is graded fairly; persistence is only a performance hint.
Error growth (weight scale 0.02, x~ℓ(0, 0.1²))
- Narrow cases are almost a residual no-op: f2 (
L=1, d=4) has max |Δ| = 1.6e-5, under atol. RMS stays ≈0.10 through f8. - At realistic width the residual is O(1) and the stream grows, but does not blow up or vanish. Perf (
L=8, d=1024): RMS 0.267 → 0.696, max |h| 1.43 → 3.48, per-layer |Δ| ≈ 1.2–1.4.
Correct fp32 vs fp64, worst share of budget:
| variant | functional | perf |
|---|---|---|
| reference / dense fp32 | 0.0115 (f10) | 0.0137 |
| split-K 32-rand / 64-rev / 128 | ≤ 0.0084 | ≤ 0.0087 |
≥73× headroom. Per-element peaks at 0.052 on perf.
Reduced precision
- TF32: first fails at f5 (1.45×); 10× on f9, 20× on f10, 22.4× on perf.
- FP16: passes f1–f8 (worst 0.88 on f8); rejected on f9 (3.2×), f10 (10.3×), perf (11.1×).
- BF16: 1.62× from f2, 93× on perf.
Injected bugs (catches / 10 functional; +perf). On perf the smallest miss is no_eps at 9.94×; skip-last is 2.7e3×, post-norm 1.9e4×, no SiLU 1.3e4×, transpose is a shape error.
- add +0.1 bias: 10 + perf
- post-norm residual / add instead of mul / no residual / transpose W: 9 + perf
- skip first layer / no RMSNorm / LayerNorm / no SiLU: 8 + perf
- skip last / only first / double last / ReLU / GELU / SiLU on both /
sumnot mean: 7 + perf - no rms_w / SiLU on up not gate / swap gate↔up: 6 + perf
- reverse layer order: 5 + perf (the example is symmetric enough that reverse matches)
- drop ε: 3 + perf (f4 via 0/0=NaN; f9, f10, perf). Missed on f1–f3 and f5–f8, where mean(h²)≫ε.
Small-width cases hide several of those bugs (f2 especially). Coverage is carried by f5+ and perf, not a blocker.
Perf (2048 × 1024 × 2816 × 8)
- I/O ≈ 294 MB (weights 277 MB). Fine on a T4; 5× ≈ 1.47 GB.
- ~283 GFLOP of FMAs, compute-bound; T4 floor ~35 ms. Not degenerate.
outputistorch.empty, not pre-filled with 123. The runner zero-fills; leavingoutputunwritten is caught on every non-zero case, but not on the all-zero-x case (the correct output is 0).
Starters: all six match the signature. JAX returns the output; CuTe uses Int32; Mojo uses max.gpu and MutAnyOrigin.
No blockers. Optional nits: pre-fill output with 123 like the other challenges; add explicit constraint upper bounds; the "hard" label is really about a persistent megakernel the grader cannot (and does not) require. As the PR body says, this is not T4-validated yet — I did not trigger that.
The all-zero-x functional case has a correct output of 0, so an unwritten buffer can pass after the runner zero-fills it. Fill every test output with 123.0, matching the other challenges.
There was a problem hiding this comment.
✅ Review bot: low risk, safe to merge. Lint and structure checks passed; no sensitive paths or grader edits; 200 lines changed.
- ID 161 is unused and the challenge is not a duplicate of #84, #93, #146, #54/#65 or #160 (a multi-layer pre-norm residual SwiGLU stack).
- challenge.py defines the 5 required methods with no init and sets name, atol, rtol, num_gpus and access_tier. It adds one private helper, _make_test_case, which is harmless.
- reference_impl asserts shapes and dtypes with no device checks and uses only standard PyTorch ops. Imports are limited to ctypes, typing, torch and the challenge base, and nothing runs at import time.
- All tensor allocations use device=self.device. The functional tests cover L=1, zero input, power-of-two and non-power-of-two sizes, and several layer counts. The two-layer example checks out by hand (about 2.6088).
- atol = rtol = 1e-4 is sensible for float32 and still rejects a skipped layer, a post-norm residual or a missing SiLU.
- The perf test needs about 294 MB (about 1.5 GB at 5x), well under the 3.2 GB limit.
- The starters are empty stubs with only the parameter comment, and the Mojo starter uses max.gpu imports and MutAnyOrigin. challenge.html matches the signature and example.
- The 'persistent megakernel' framing is not enforced, but correctness-only grading is stated openly and the task is still a non-trivial GPU exercise.
|
@claude Please validate this new challenge on T4. Follow CLAUDE.md: verify every checklist item, write an independent CUDA solution in Re-run on head |
1 similar comment
|
@claude Please validate this new challenge on T4. Follow CLAUDE.md: verify every checklist item, write an independent CUDA solution in Re-run on head |
T4 validation on
|
| Step | Result |
|---|---|
--action run (example test) |
✅ Passed on T4 |
--action submit (functional + performance) |
Runner didn't start, you were not charged; this is not a problem with your code after "T4 is taking a while to start / slow GPU host" retries |
I stopped at 4 of the 5 runs allowed per session, so I have no functional-test or performance timings yet. Please re-trigger once T4 capacity recovers.
Checklist review
- challenge.py: 5 attributes, no
__init__, all 5 methods present. Shape and dtype asserts only, standard PyTorch ops that also run on XLA,device=self.devicethroughout. The functional tests cover L=1, zero input, power-of-2 and non-power-of-2 sizes, and L up to 8. The example is checked against independently computed values; I re-derived 2.608823 by hand. The perf case is about 294 MB, well within 5× of 16 GB. ✅ - Starters: all 6 present. Each has exactly one parameter comment ("device pointers" with no parenthetical, as required for hard), and JAX has
# return output tensor directly. They are emptysolvebodies, and Mojo uses the 1.1 style. ✅ - Lint: black, isort, flake8, clang-format and the whitespace hooks pass. The
mojo formathook could not run becausemojois not installed in the CI runner. - Nit (challenge.html): CLAUDE.md asks for LaTeX
\begin{bmatrix}for 2D/3D example data, but the example uses a<pre>block. The 3D weight tensors are awkward in bmatrix, so this is optional.
🤖 Generated with Claude Code
|
@claude Please validate this new challenge on T4. Follow CLAUDE.md: verify every checklist item, write an independent CUDA solution in Re-run on head |
T4 validation on
|
| Step | Result |
|---|---|
--action run (example) |
✅ Test passed |
--action submit (all functional tests + performance test) |
✅ All tests passed (run twice, passed both times) |
Timings: not available. The submit API's final message has only submissionId, status, type and output, and contains no runtime or GFLOPS fields. I checked this by logging the raw WebSocket message on the second submit (submission eb5755da-f94f-4855-a597-21e5f470df24). Timings would need to come from the platform UI.
Solution used (not committed): independent CUDA, non-persistent. It copies x into output, then runs three kernels per layer:
- RMSNorm, one block per row.
- A fused gate/up 64×64 tiled SGEMM with SiLU⊙up.
- A down-projection tiled SGEMM that accumulates into
output.
All loads are bounds-checked, so the non-power-of-two tails (M=30/100, d_model=24/60, d_ffn=40/84) are exercised. The earlier run on this PR confirmed that a cooperative-groups persistent version also passes.
Checklist (CLAUDE.md)
- challenge.html: starts with
<p>, has<h2>sections and a dark SVG. The example matchesgenerate_example_test(); I re-derived 2.608823 = 1 + SiLU(√2)·√2 with ε folded in. The perf bullet matchesM=2,048,d_model=1,024,d_ffn=2,816,L=8. ✅ - challenge.py: 5 attributes and no
__init__. Asserts check only shape and dtype; ops are standard and XLA-safe; every allocation usesdevice=self.device. 10 functional cases, including zerox(now caught by the 123.0 pre-fill),L=1/5/8 and non-power-of-two shapes. The independent check on the example runs. The perf working set is ~294 MB. ✅ - Starters: all 6 present, one parameter comment each with the hard-level wording, JAX has
# return output tensor directly, all empty. ✅ - Lint: black, isort, flake8, clang-format and the whitespace hooks pass.
mojo formatwas not run because the runner has nomojobinary.
Non-blocking nits (unchanged from before): the example uses <pre> rather than bmatrix for 2D/3D data, and no case has d_model=1 or d_ffn=1.
🤖 Generated with Claude Code
New hard challenge #161 Persistent Multi-Layer MLP: a stack of
Lpre-norm residual SwiGLU MLP layers, intended as a persistent megakernel — one launch that keeps the grid alive and uses grid-wide sync (e.g. cooperative groups) between layers, rather than launching a kernel per layer.It is distinct from #84 (single-layer SwiGLU, no RMSNorm and no residual), #54 / #65 (element-wise SwiGLU / GeGLU), #146 (fused gate-up-SiLU only), and #93 / #160 (full decoder block, not a multi-layer MLP stack).
Behavior
l = 0 … L-1: RMS-normalize the residual stream (ε = 1e-5, no bias), apply SwiGLU (SiLU(z W_gate[l]) ⊙ z W_up[l], thenW_down[l]), and add back to the residual stream.rms_wis[L, d_model],W_gate/W_upare[L, d_model, d_ffn],W_downis[L, d_ffn, d_model]. Matrices are row-major(in, out)and applied asz × W[l].Tests
[[2.608823, 0], [0, 2.608823]].L = 1(pre-norm residual SwiGLU, unlike [BUG] Question index issue, change 53_swiglu -> 54_swiglu #84); zero input (output must stay 0 for anyL); power-of-two sizes with 4 and 8 layers; non-power-of-two shapes includingL = 5; and two realistic prefill-sized stacks.M= 2,048,d_model= 1,024,d_ffn= 2,816,L= 8. Float32 working set ≈ 294 MB, about 1.47 GB at 5×, inside a 16 GB T4. No bf16 or WMMA.Tolerance
atol = rtol = 1e-4. The float32 reference differs from a float64 recomputation by well under 1e-5 on the example and on an 8-layer mid-size case. That budget still rejects a skipped layer, a post-norm residual, or a missing SiLU.Verified locally
reference_implon the example and several functional cases; a hand expansion of the example also matches.validate_challenges.pypass.@claudevalidation comment from a maintainer.