Skip to content

Add Persistent Multi-Layer MLP challenge (hard #161) - #386

Open
kunal-mansukhani wants to merge 2 commits into
mainfrom
add-challenge-161-persistent-multi-layer-mlp
Open

kunal-mansukhani wants to merge 2 commits into
mainfrom
add-challenge-161-persistent-multi-layer-mlp

Conversation

@kunal-mansukhani

Copy link
Copy Markdown
Contributor

New hard challenge #161 Persistent Multi-Layer MLP: a stack of L pre-norm residual SwiGLU MLP layers, intended as a persistent megakernel — one launch that keeps the grid alive and uses grid-wide sync (e.g. cooperative groups) between layers, rather than launching a kernel per layer.

It is distinct from #84 (single-layer SwiGLU, no RMSNorm and no residual), #54 / #65 (element-wise SwiGLU / GeGLU), #146 (fused gate-up-SiLU only), and #93 / #160 (full decoder block, not a multi-layer MLP stack).

Behavior

  • For each layer l = 0 … L-1: RMS-normalize the residual stream (ε = 1e-5, no bias), apply SwiGLU (SiLU(z W_gate[l]) ⊙ z W_up[l], then W_down[l]), and add back to the residual stream.
  • Weights are stored with a leading layer axis: rms_w is [L, d_model], W_gate / W_up are [L, d_model, d_ffn], W_down is [L, d_ffn, d_model]. Matrices are row-major (in, out) and applied as z × W[l].
  • The statement frames a single persistent launch as the intended approach; the judge still checks numerical correctness, not launch count.

Tests

  • 10 functional cases, including a two-layer hand-computed example (rank-1 SwiGLU on coordinate 0, then on coordinate 1) checked against an independent per-row float64 loop. Output is [[2.608823, 0], [0, 2.608823]].
  • Targeted cases: L = 1 (pre-norm residual SwiGLU, unlike [BUG] Question index issue, change 53_swiglu -> 54_swiglu #84); zero input (output must stay 0 for any L); power-of-two sizes with 4 and 8 layers; non-power-of-two shapes including L = 5; and two realistic prefill-sized stacks.
  • Perf case: M = 2,048, d_model = 1,024, d_ffn = 2,816, L = 8. Float32 working set ≈ 294 MB, about 1.47 GB at 5×, inside a 16 GB T4. No bf16 or WMMA.

Tolerance

  • atol = rtol = 1e-4. The float32 reference differs from a float64 recomputation by well under 1e-5 on the example and on an 8-layer mid-size case. That budget still rejects a skipped layer, a post-norm residual, or a missing SiLU.

Verified locally

  • Independent per-row float64 loop matches reference_impl on the example and several functional cases; a hand expansion of the example also matches.
  • black, isort, flake8, clang-format and validate_challenges.py pass.
  • Not T4-verified yet: that needs an @claude validation comment from a maintainer.

Stack of pre-norm residual SwiGLU MLP layers in one persistent megakernel,
using grid-wide sync between layers instead of a launch per layer.
github-actions[bot]
github-actions Bot previously approved these changes Nov 11, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Review bot: low risk, safe to merge. Lint and structure checks passed; no sensitive paths or grader edits; 198 lines changed.

  • ID 161 is not used by any existing challenge. It is distinct from #84, #54, #65, #146, #93 and the other listed challenges, because it stacks L pre-norm residual SwiGLU layers.
  • It is non-trivial: a multi-layer fused MLP with grid-wide synchronization is a real GPU exercise.
  • challenge.py has the 5 required methods, no init override, and name/atol/rtol/num_gpus/access_tier set. The extra _make_test_case helper is harmless.
  • Imports are only ctypes, typing, torch, torch.nn.functional and core.challenge_base. There is no network, file, subprocess or import-time code.
  • reference_impl asserts shapes and dtypes, has no device checks, and uses standard PyTorch ops only.
  • All generate_* allocations use device=self.device.
  • Functional tests cover L=1, zero input, power-of-two and non-power-of-two sizes, L=5, and large stacks. The hand-computed example checks out: 1 + 1.609 ≈ 2.6088.
  • atol=rtol=1e-4 is reasonable for float32 and would still reject a skipped layer, a post-norm residual or a missing SiLU.
  • The performance test uses about 294 MB, well under the 3.2 GB limit.
  • Starters are empty stubs with only the parameter comment. The Mojo starter uses max.gpu imports and MutAnyOrigin, and challenge.html matches the signature and example.

@kunal-mansukhani

Copy link
Copy Markdown
Contributor Author

@claude Please validate this new challenge on T4. Follow CLAUDE.md: verify every checklist item, write an independent CUDA solution in solution/solution.cu (do not commit it), and run python scripts/run_challenge.py challenges/hard/161_persistent_multi_layer_mlp --language cuda --action run then --action submit. Report whether the example, functional tests, and performance test pass.

@claude

claude Bot commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

T4 validation: Persistent Multi-Layer MLP (#161)

Result: passes on T4. I wrote a CUDA solution as a single persistent kernel: cudaLaunchCooperativeKernel plus cg::this_grid().sync(), with three phases per layer (RMSNorm, then fused gate/up + SiLU, then down-proj + residual) and 64×64 SIMT GEMM tiles.

  • --action run (example): ✓ passed
  • --action submit (all functional tests + the performance test): ✓ all tests passed

This also confirms the intended cooperative-groups approach compiles and runs on the platform without -rdc. The solution file was not committed.

Checklist (CLAUDE.md)

  • challenge.html: starts with <p>, has the <h2> sections, first example matches generate_example_test() (I re-derived 2.608823 by hand), uses one <pre> format, perf bullet matches M=2,048, d_model=1,024, d_ffn=2,816, L=8, includes a dark SVG ✓
  • challenge.py: 5 class attributes and no __init__; asserts check only shape and dtype; standard ops only (XLA-safe); every allocation uses device=self.device; 10 functional cases including zeros, non-power-of-2 tails and L = 1/5/8; the hand-computed check runs; perf working set is ~294 MB ✓
  • Starters: all 6 present, exactly one parameter comment each with the medium/hard wording, JAX has # return output tensor directly, no solving logic ✓
  • Lint: black / isort / flake8 / clang-format pass. mojo format couldn't run because the runner has no mojo binary; this is an environment issue, not a code issue.

Optional nits (non-blocking)

  • The example is symmetric, so applying the layers in reverse order gives the same output for it. The random multi-layer cases do catch a reversed order.
  • No case has d_model = 1 or d_ffn = 1. The constraints allow both.

@shxjames shxjames left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 4535833. Independent fp64 CPU stack (pre-norm RMSNorm, SiLU-SwiGLU, residual add). Margins are max-scaled per output (atol + rtol·max|ref|, atol = rtol = 1e-4).

Spec / reference

  • Clear and matches the reference: no bias, ε = 1e-5, pre-norm residual (add uses h, not the normalized values), layers 0…L−1 in order, z × W[l] with (in, out) layouts (W_gate/W_up are [L, d_model, d_ffn], W_down is [L, d_ffn, d_model]). Last layer uses the same residual + SiLU formula; output = h^{(L)}.
  • The example closed form is 1 + σ(r)·r² with r = 1/sqrt(1/2+ε) = 2.60882274; the reference is 2.60882258.
  • Every dict passes all 10 params, including output. The reference leaves the inputs untouched.
  • Persistence: the write-up correctly says the judge checks numerics, not launch count. A per-layer launch is graded fairly; persistence is only a performance hint.

Error growth (weight scale 0.02, x~ℓ(0, 0.1²))

  • Narrow cases are almost a residual no-op: f2 (L=1, d=4) has max |Δ| = 1.6e-5, under atol. RMS stays ≈0.10 through f8.
  • At realistic width the residual is O(1) and the stream grows, but does not blow up or vanish. Perf (L=8, d=1024): RMS 0.267 → 0.696, max |h| 1.43 → 3.48, per-layer |Δ| ≈ 1.2–1.4.

Correct fp32 vs fp64, worst share of budget:

variant functional perf
reference / dense fp32 0.0115 (f10) 0.0137
split-K 32-rand / 64-rev / 128 ≤ 0.0084 ≤ 0.0087

≥73× headroom. Per-element peaks at 0.052 on perf.

Reduced precision

  • TF32: first fails at f5 (1.45×); 10× on f9, 20× on f10, 22.4× on perf.
  • FP16: passes f1–f8 (worst 0.88 on f8); rejected on f9 (3.2×), f10 (10.3×), perf (11.1×).
  • BF16: 1.62× from f2, 93× on perf.

Injected bugs (catches / 10 functional; +perf). On perf the smallest miss is no_eps at 9.94×; skip-last is 2.7e3×, post-norm 1.9e4×, no SiLU 1.3e4×, transpose is a shape error.

  • add +0.1 bias: 10 + perf
  • post-norm residual / add instead of mul / no residual / transpose W: 9 + perf
  • skip first layer / no RMSNorm / LayerNorm / no SiLU: 8 + perf
  • skip last / only first / double last / ReLU / GELU / SiLU on both / sum not mean: 7 + perf
  • no rms_w / SiLU on up not gate / swap gate↔up: 6 + perf
  • reverse layer order: 5 + perf (the example is symmetric enough that reverse matches)
  • drop ε: 3 + perf (f4 via 0/0=NaN; f9, f10, perf). Missed on f1–f3 and f5–f8, where mean(h²)≫ε.

Small-width cases hide several of those bugs (f2 especially). Coverage is carried by f5+ and perf, not a blocker.

Perf (2048 × 1024 × 2816 × 8)

  • I/O ≈ 294 MB (weights 277 MB). Fine on a T4; 5× ≈ 1.47 GB.
  • ~283 GFLOP of FMAs, compute-bound; T4 floor ~35 ms. Not degenerate.
  • output is torch.empty, not pre-filled with 123. The runner zero-fills; leaving output unwritten is caught on every non-zero case, but not on the all-zero-x case (the correct output is 0).

Starters: all six match the signature. JAX returns the output; CuTe uses Int32; Mojo uses max.gpu and MutAnyOrigin.

No blockers. Optional nits: pre-fill output with 123 like the other challenges; add explicit constraint upper bounds; the "hard" label is really about a persistent megakernel the grader cannot (and does not) require. As the PR body says, this is not T4-validated yet — I did not trigger that.

The all-zero-x functional case has a correct output of 0, so an unwritten
buffer can pass after the runner zero-fills it. Fill every test output
with 123.0, matching the other challenges.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Review bot: low risk, safe to merge. Lint and structure checks passed; no sensitive paths or grader edits; 200 lines changed.

  • ID 161 is unused and the challenge is not a duplicate of #84, #93, #146, #54/#65 or #160 (a multi-layer pre-norm residual SwiGLU stack).
  • challenge.py defines the 5 required methods with no init and sets name, atol, rtol, num_gpus and access_tier. It adds one private helper, _make_test_case, which is harmless.
  • reference_impl asserts shapes and dtypes with no device checks and uses only standard PyTorch ops. Imports are limited to ctypes, typing, torch and the challenge base, and nothing runs at import time.
  • All tensor allocations use device=self.device. The functional tests cover L=1, zero input, power-of-two and non-power-of-two sizes, and several layer counts. The two-layer example checks out by hand (about 2.6088).
  • atol = rtol = 1e-4 is sensible for float32 and still rejects a skipped layer, a post-norm residual or a missing SiLU.
  • The perf test needs about 294 MB (about 1.5 GB at 5x), well under the 3.2 GB limit.
  • The starters are empty stubs with only the parameter comment, and the Mojo starter uses max.gpu imports and MutAnyOrigin. challenge.html matches the signature and example.
  • The 'persistent megakernel' framing is not enforced, but correctness-only grading is stated openly and the task is still a non-trivial GPU exercise.

@kunal-mansukhani

Copy link
Copy Markdown
Contributor Author

@claude Please validate this new challenge on T4. Follow CLAUDE.md: verify every checklist item, write an independent CUDA solution in solution/solution.cu (do not commit it), and run python scripts/run_challenge.py challenges/hard/161_persistent_multi_layer_mlp --language cuda --action run then --action submit. Report whether the example, functional tests, and performance test pass.

Re-run on head ad9dbbc95c89fd34d96dcaad2abbe7084fefe5b1 after the latest review fixes. Please post the T4 result (pass/fail with timings) as a PR comment when done.

1 similar comment
@kunal-mansukhani

Copy link
Copy Markdown
Contributor Author

@claude Please validate this new challenge on T4. Follow CLAUDE.md: verify every checklist item, write an independent CUDA solution in solution/solution.cu (do not commit it), and run python scripts/run_challenge.py challenges/hard/161_persistent_multi_layer_mlp --language cuda --action run then --action submit. Report whether the example, functional tests, and performance test pass.

Re-run on head ad9dbbc95c89fd34d96dcaad2abbe7084fefe5b1 after the latest review fixes. Please post the T4 result (pass/fail with timings) as a PR comment when done.

@claude

claude Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

T4 validation on ad9dbbc (partial: only --action run finished)

Solution used: an independent CUDA solution, not committed. Per layer it launches three kernels: RMSNorm, a fused gate/up tiled SGEMM with SiLU and the product, and a down-projection tiled SGEMM that adds into the residual in output. It is deliberately not persistent, since the grader checks correctness only.

Step Result
--action run (example test) ✅ Passed on T4
--action submit (functional + performance) ⚠️ Not run. All 3 attempts ended with Runner didn't start, you were not charged; this is not a problem with your code after "T4 is taking a while to start / slow GPU host" retries

I stopped at 4 of the 5 runs allowed per session, so I have no functional-test or performance timings yet. Please re-trigger once T4 capacity recovers.

Checklist review

  • challenge.py: 5 attributes, no __init__, all 5 methods present. Shape and dtype asserts only, standard PyTorch ops that also run on XLA, device=self.device throughout. The functional tests cover L=1, zero input, power-of-2 and non-power-of-2 sizes, and L up to 8. The example is checked against independently computed values; I re-derived 2.608823 by hand. The perf case is about 294 MB, well within 5× of 16 GB. ✅
  • Starters: all 6 present. Each has exactly one parameter comment ("device pointers" with no parenthetical, as required for hard), and JAX has # return output tensor directly. They are empty solve bodies, and Mojo uses the 1.1 style. ✅
  • Lint: black, isort, flake8, clang-format and the whitespace hooks pass. The mojo format hook could not run because mojo is not installed in the CI runner.
  • Nit (challenge.html): CLAUDE.md asks for LaTeX \begin{bmatrix} for 2D/3D example data, but the example uses a <pre> block. The 3D weight tensors are awkward in bmatrix, so this is optional.

🤖 Generated with Claude Code

@kunal-mansukhani

Copy link
Copy Markdown
Contributor Author

@claude Please validate this new challenge on T4. Follow CLAUDE.md: verify every checklist item, write an independent CUDA solution in solution/solution.cu (do not commit it), and run python scripts/run_challenge.py challenges/hard/161_persistent_multi_layer_mlp --language cuda --action run then --action submit. Report whether the example, functional tests, and performance test pass.

Re-run on head ad9dbbc95c89fd34d96dcaad2abbe7084fefe5b1 after the latest review fixes. Please post the T4 result (pass/fail with timings) as a PR comment when done.

@claude

claude Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

T4 validation on ad9dbbc: ✅ passes

Step Result
--action run (example) ✅ Test passed
--action submit (all functional tests + performance test) ✅ All tests passed (run twice, passed both times)

Timings: not available. The submit API's final message has only submissionId, status, type and output, and contains no runtime or GFLOPS fields. I checked this by logging the raw WebSocket message on the second submit (submission eb5755da-f94f-4855-a597-21e5f470df24). Timings would need to come from the platform UI.

Solution used (not committed): independent CUDA, non-persistent. It copies x into output, then runs three kernels per layer:

  1. RMSNorm, one block per row.
  2. A fused gate/up 64×64 tiled SGEMM with SiLU⊙up.
  3. A down-projection tiled SGEMM that accumulates into output.

All loads are bounds-checked, so the non-power-of-two tails (M=30/100, d_model=24/60, d_ffn=40/84) are exercised. The earlier run on this PR confirmed that a cooperative-groups persistent version also passes.

Checklist (CLAUDE.md)

  • challenge.html: starts with <p>, has <h2> sections and a dark SVG. The example matches generate_example_test(); I re-derived 2.608823 = 1 + SiLU(√2)·√2 with ε folded in. The perf bullet matches M=2,048, d_model=1,024, d_ffn=2,816, L=8. ✅
  • challenge.py: 5 attributes and no __init__. Asserts check only shape and dtype; ops are standard and XLA-safe; every allocation uses device=self.device. 10 functional cases, including zero x (now caught by the 123.0 pre-fill), L=1/5/8 and non-power-of-two shapes. The independent check on the example runs. The perf working set is ~294 MB. ✅
  • Starters: all 6 present, one parameter comment each with the hard-level wording, JAX has # return output tensor directly, all empty. ✅
  • Lint: black, isort, flake8, clang-format and the whitespace hooks pass. mojo format was not run because the runner has no mojo binary.

Non-blocking nits (unchanged from before): the example uses <pre> rather than bmatrix for 2D/3D data, and no case has d_model=1 or d_ffn=1.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants