Repository navigation
Add CLIP Contrastive Loss challenge (hard #162) - #387
claude[bot] wants to merge 1 commit into
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
✅ Review bot: low risk, safe to merge. Lint and structure checks passed; no sensitive paths or grader edits; 141 lines changed.
- Numeric ID 162 is unused by any existing challenge. The challenge is distinct from #129, #128, #141, #25 and #150 because it needs a two-direction (row and column) LSE merge from one GEMM.
- challenge.py defines the 5 required methods with no init override and sets name, atol, rtol, num_gpus and access_tier. The helper methods _pack, _make and _randn are harmless.
- Imports are limited to ctypes, typing, torch and the challenge base. There is no network, file or subprocess access and no import-time code.
- reference_impl asserts shapes and dtypes and uses only standard PyTorch ops. All allocations in generate_* use device=self.device.
- Functional tests cover N=1, logit_scale=0, wildly varying row norms, asymmetric logits, overflow-range logits, tail tiles, duplicate columns, and power-of-two and non-power-of-two sizes. The 1e-3 tolerances are sensible for a loss of about 3.
- Performance inputs and outputs total about 64 MB, well under the memory limit.
- challenge.html is consistent with the signature, and I re-checked the Example 1 numbers (1.7325); they are correct.
- All six starters are empty stubs with only the parameter comment. The Mojo starter uses max.gpu imports and MutAnyOrigin.
- The PR body is cut off mid-sentence, but the diff is complete and fully judgeable.
shxjames
left a comment
There was a problem hiding this comment.
Reviewed f4bfa52. Spec and reference agree; no correctness blocker.
Spec / reference. Every row is L2-normalized with no epsilon (constraints already require a nonzero L2). S = logit_scale * x̂ ŷᵀ: logit_scale is the already-scaled multiplier in [0, 100], not exp(log_scale) and not a temperature divisor. The loss is ½ (mean of row CE + mean of column CE) with diagonal targets. The only output is a scalar loss of shape (1,); no gradients. On a 1-element tensor the max-scaled grader is just |\u0394| ≤ 1e-3 · (1 + |loss|), which is a meaningful check (budget ≈ 2e-3–1.5e-2 on the non-trivial cases).
An independent fp64 CPU reimplementation matches the reference. The HTML example (row losses 0.2633 / 3.6270, column 0.9130 / 2.1269, loss 1.7325) matches both (fp64 1.73254570, ref 1.73254561).
Headroom (share of atol + rtol · |ref|). Correct fp32, F.cross_entropy, and tiled online-LSE (256 / 128 / 64-reversed) stay at ≤ 2.7e-4 of budget on every functional case and ≤ 4.9e-5 on perf (≥ 3700×). Reduction order is not a problem.
TF32 / FP16 / BF16 all pass, though: worst shares are 0.055 / 0.056 / 0.177 on the N=48 asymmetric case, and 0.00034 / 0.0017 / 0.016 on perf. Because the output is one averaged scalar, 1e-3 is loose enough that reduced-precision shortcuts succeed everywhere. Optional: tighten to ~5e-5 (correct fp32 would still have ~190× room, and that same case would reject all three). Not a correctness blocker if that is acceptable.
Injected bugs. Missing L2, L2 along dim 0, divide by ‖x‖², divide by / omit / exp of logit_scale, off-by-one or zero targets, no /2, sum instead of mean, and uniform CE are all caught, usually by a large margin (N=1 and logit_scale=0 miss several by construction). One-direction-only is missed on the nearly-symmetric cases, including perf (0.63 of budget; i2t/t2i gap 0.003 nats), but is caught hard on the designed asymmetric tests (N=48: gap 3.37 nats, 257×; N=255: gap 7.11 nats, 234×) and the example (78×). A missing max-subtraction in fp32 overflows only on the clustered scale=100 case (logits ≈ 100); every other case, including perf (diag in [1.8, 36]), stays finite and matches — so only that one test is doing that job. Adding 1e-6 (or 1e-12) to the L2 denom is never caught (worst 0.0015 of budget), which is fine given the spec has no eps.
Perf. N=16384, D=512, scale=100: loss is 1.4535 (the PR text says ~3.2; same order, just stale). I/O 67 MB; materializing S is 1.07 GB, still fine on a T4. GEMM is ~275 GFLOP, theoretical T4 fp32 floor ~34 ms, compute-bound as claimed. Every test dict includes the loss buffer (zeros, not a 123 prefill; an unwritten 0 fails every case except N=1, where the expected loss is 0).
All six starters match the signature; JAX returns the output. CI is green (Lint Python, Lint C++/CUDA, Check Mojo, check-index, checks, review). I have not approved or triggered a T4 run; the author reports a fused CUDA solution already passed --action submit on a T4.
Summary
Adds CLIP Contrastive Loss (hard #162): the symmetric InfoNCE loss used to train CLIP/ALIGN-style image–text models (and dual-encoder retrievers). Inputs are
image_embedsandtext_embeds, eachN × Dwith arbitrary row norms, pluslogit_scale. Each row is L2-normalized,S = logit_scale · x̂ ŷᵀis formed, and the output is the mean of the row-softmax (image→text) and column-softmax (text→image) cross-entropies, with the diagonal as targets.What a solver learns
Stile updates the running (max, sum-exp) state for its rows and its columns.1/‖x_i‖and1/‖y_j‖once and apply them as rank-1 scales to each accumulator tile.Novelty vs. the closest existing challenges
SplusSᵀ) is the key design decision. The row normalization folded into the epilogue is also new.Naive vs. optimized
N×NS(1 GB at the benchmark size), then run a row-LSE kernel and a column-LSE kernel. A naive column-LSE kernel readsSwith strideN, which is uncoalesced. Alternatively, a "fused" version that recomputes the GEMM once per direction doubles the FLOPs.Sthat never writes it. Each block computes a 64×64 (or larger) tile from shared-memory-stagedxandy, appliesscale · inv_x[i] · inv_y[j], and reduces the tile into row partials (warp shuffles) and column partials (shared memory). A small merge kernel combines theN/BMpartials per row and per column, then a deterministic final reduction computes the mean.How the benchmark exposes it
Performance test:
N = 16,384,D = 512,logit_scale = 100. The GEMM is about 275 GFLOP, so a good solution is compute-bound on the T4's FP32 units.Sadds a 1 GB write plus two 1 GB reads (the column read is strided unless transposed), and memory use grows as O(N²). The fused version only touches the 64 MB of inputs plusO(N²/BM)partials (about 2 × 256 × 16,384 float2 ≈ 67 MB at BM = 64).text = image + 5·noise), so the loss is O(1) nats (~3.2) and off-diagonal logits really contribute (CLIP-like rather than degenerate).logit_scale = 100is CLIP's clamp value, so logits reach tens of nats and need a correct running max.Targeted tests (mistakes caught)
I checked locally that each mistake below differs from the reference by far more than the tolerance in at least one test:
Sis symmetric: an asymmetric case (near-identical image rows, diverse text rows; 1.68 nat gap), plus the example (0.21) and N=255 (3.56). Also confirmed on a T4: a row-only CUDA variant fails the example with 1.9451 vs 1.7325.expoverflows float32 while the loss stays non-trivial (3.48).logit_scale: the large-diagonal cases (N=512, 1000, 4096).logit_scale = 0is the zero-input case (loss must be exactly log N).Tolerance: atol = rtol = 1e-3. The loss is a mean of LSE values of magnitude ≤ ~10, built from unit-vector dot products with |logit| ≤ 100. FP32 accumulation-order error is ~1e-5, and every mistake above is off by ≥ 0.2 nats.
Validation
scripts/run_challenge.py --accelerator T4with a fused tiled CUDA solution:--action runpassed and--action submitpassed all functional tests plus the performance test. The solution file is not committed.pre-commit run --all-filesandscripts/validate_challenges.pypass.<p>, uses<h2>sections, includes an SVG (#222background) and LaTeX matrix examples, and has the performance bullet.__init__, and all allocations usedevice=self.device.🤖 Generated with Claude Code