Repository navigation
JAXBench: add level3 backward-pass workloads from Tokamax - #146
cccathygao wants to merge 1 commit into
Conversation
30 workloads under JAXBench/benchmark/level3, one baseline.py each, built from 16 Tokamax ops (openxla/tokamax @ b4b6eb3): - Track 1 (16 workloads): pure-JAX forward reference; workload() returns the gradients via jax.vjp. - Track 2 (14 workloads): the same plus Tokamax's forward-only Pallas TPU kernel, inlined as forward_pallas(). Checked on CPU only (Pallas interpret mode); nothing has run on a TPU.
There was a problem hiding this comment.
Code Review
This pull request introduces pure-JAX reference implementations and benchmark harnesses for several JAXBench level 3 backward pass operators, covering various attention mechanisms, normalization layers, and matrix multiplications. The feedback highlights critical performance and correctness issues in the added baselines. Specifically, in the KDA operators, the nested jax.lax.fori_loop should be refactored using jax.vmap and jax.lax.scan to allow parallelization and support reverse-mode automatic differentiation (jax.vjp). Additionally, in the linear softmax cross-entropy operators, the memory-intensive creation of a full one-hot tensor should be replaced with advanced indexing to directly extract target log-probabilities.
| states = jnp.zeros((batch, 1, heads, key_dim, value_dim), dtype=acc_dtype) | ||
| output_h = jnp.zeros((heads, batch, seq_len, value_dim), dtype=acc_dtype) | ||
|
|
||
| def step_token(h, b, t, carry): | ||
| states, output_h = carry | ||
| previous_state = states[b, 0, h] | ||
| state = previous_state * jnp.exp(g_h[h, b, t])[:, None] | ||
| prediction = k_h[h, b, t] @ state | ||
| residual = v_h[h, b, t] - prediction | ||
| new_state = state + ( | ||
| beta_h[h, b, t] * k_h[h, b, t] | ||
| )[:, None] * residual[None, :] | ||
| out_t = q_h[h, b, t] @ new_state | ||
| output_h = output_h.at[h, b, t].set(out_t) | ||
| states = states.at[b, 0, h].set(new_state) | ||
| return states, output_h | ||
|
|
||
| def step_head(h, carry): | ||
| states, output_h = carry | ||
|
|
||
| def body_b(b, b_carry): | ||
| def body_t(t, t_carry): | ||
| return step_token(h, b, t, t_carry) | ||
|
|
||
| return jax.lax.fori_loop(0, seq_len, body_t, b_carry) | ||
|
|
||
| states, output_h = jax.lax.fori_loop( | ||
| 0, batch, body_b, (states, output_h) | ||
| ) | ||
| return states, output_h | ||
|
|
||
| states, output_h = jax.lax.fori_loop( | ||
| 0, heads, step_head, (states, output_h) | ||
| ) |
There was a problem hiding this comment.
Using nested jax.lax.fori_loop to sequentially iterate over heads and batch elements is highly inefficient on accelerators (TPU/GPU) as it prevents parallelization. Furthermore, fori_loop is implemented using while_loop under the hood in JAX, which does not support reverse-mode automatic differentiation (jax.vjp), causing workload to fail at runtime.\n\nWe can rewrite this using jax.vmap to parallelize over heads and batch dimensions, and jax.lax.scan to efficiently propagate the recurrent state over the sequence dimension. This is both fully differentiable and highly performant.
def step_fn(state, inputs):\n q_t, k_t, v_t, g_t, beta_t = inputs\n state = state * jnp.exp(g_t)[:, None]\n prediction = k_t @ state\n residual = v_t - prediction\n state = state + (beta_t * k_t)[:, None] * residual[None, :]\n out_t = q_t @ state\n return state, out_t\n\n def scan_fn(q_hb, k_hb, v_hb, g_hb, beta_hb):\n s0 = jnp.zeros((key_dim, value_dim), dtype=acc_dtype)\n _, out_hb = jax.lax.scan(step_fn, s0, (q_hb, k_hb, v_hb, g_hb, beta_hb))\n return out_hb\n\n vmap_fn = jax.vmap(jax.vmap(scan_fn, in_axes=0), in_axes=0)\n output_h = vmap_fn(q_h, k_h, v_h, g_h, beta_h)| """KDA forward pass: explicit token-by-token delta-rule recurrence. | ||
|
|
||
| Args: | ||
| q: Queries, `[H, B, T, K]`. L2-normalised over `K` inside the op. | ||
| k: Keys, `[H, B, T, K]`. L2-normalised over `K` inside the op. | ||
| v: Values, `[H, B, T, V]`. | ||
| g: Raw per-channel gate input, `[H, B, T, K]`. The log-decay is | ||
| `-5 * sigmoid(exp(a_log) * (g + delta_time_bias))`. | ||
| beta: Per-token delta-rule learning rate, `[H, B, T]`, values in `[0, 1]`. | ||
| a_log: Per-head log decay rate, `[H]`. | ||
| delta_time_bias: Per-head, per-key-channel gate bias, `[H * K]`. | ||
|
|
||
| Returns: | ||
| The output, `[H, B, T, V]`, in the dtype of `q`. The state starts at zero | ||
| and the final state is not returned. Queries are scaled by `K ** -0.5`. | ||
| """ | ||
| heads, batch, seq_len, key_dim = q.shape | ||
| value_dim = v.shape[-1] | ||
| acc_dtype = _accumulator_dtype(q.dtype) | ||
| output_dtype = q.dtype | ||
| scale = key_dim**-0.5 | ||
|
|
||
| g = _activate_gate( | ||
| g, a_log=a_log, delta_time_bias=delta_time_bias, lower_bound=_LOWER_BOUND | ||
| ) | ||
| q_h = _l2_normalize(q, acc_dtype) | ||
| k_h = _l2_normalize(k, acc_dtype) | ||
|
|
||
| q_h = q_h * scale | ||
| v_h = v.astype(acc_dtype) | ||
| g_h = g.astype(acc_dtype) | ||
| beta_h = beta.astype(acc_dtype) | ||
|
|
||
| # One recurrent state per batch element and head. |
There was a problem hiding this comment.
Using nested jax.lax.fori_loop to sequentially iterate over heads and batch elements is highly inefficient on accelerators (TPU/GPU) as it prevents parallelization. Furthermore, fori_loop is implemented using while_loop under the hood in JAX, which does not support reverse-mode automatic differentiation (jax.vjp), causing workload to fail at runtime.\n\nWe can rewrite this using jax.vmap to parallelize over heads and batch dimensions, and jax.lax.scan to efficiently propagate the recurrent state over the sequence dimension. This is both fully differentiable and highly performant.
def step_fn(state, inputs):\n q_t, k_t, v_t, g_t, beta_t = inputs\n state = state * jnp.exp(g_t)[:, None]\n prediction = k_t @ state\n residual = v_t - prediction\n state = state + (beta_t * k_t)[:, None] * residual[None, :]\n out_t = q_t @ state\n return state, out_t\n\n def scan_fn(q_hb, k_hb, v_hb, g_hb, beta_hb):\n s0 = jnp.zeros((key_dim, value_dim), dtype=acc_dtype)\n _, out_hb = jax.lax.scan(step_fn, s0, (q_hb, k_hb, v_hb, g_hb, beta_hb))\n return out_hb\n\n vmap_fn = jax.vmap(jax.vmap(scan_fn, in_axes=0), in_axes=0)\n output_h = vmap_fn(q_h, k_h, v_h, g_h, beta_h)| labels_one_hot = jax.nn.one_hot(labels, num_classes=w.shape[1], dtype=x.dtype) | ||
| loss = -labels_one_hot * log_probs | ||
| loss_per_sample = jnp.sum(loss, axis=-1, dtype=jnp.float32) |
There was a problem hiding this comment.
Creating a full one-hot tensor of shape (B, V) (where V is the vocabulary size, e.g., 128,256) and performing element-wise multiplication is highly inefficient in terms of both memory and computation. Since the loss is only non-zero at the target label indices, we can use advanced indexing to extract the target log-probabilities directly. This avoids allocating the large one-hot tensor and performing redundant operations.
| labels_one_hot = jax.nn.one_hot(labels, num_classes=w.shape[1], dtype=x.dtype) | |
| loss = -labels_one_hot * log_probs | |
| loss_per_sample = jnp.sum(loss, axis=-1, dtype=jnp.float32) | |
| loss_per_sample = -log_probs[jnp.arange(x.shape[0]), labels].astype(jnp.float32) |
|
|
||
| # ============================================================================== | ||
| # Forward reference |
There was a problem hiding this comment.
Creating a full one-hot tensor of shape (B, V) (where V is the vocabulary size, e.g., 128,256) and performing element-wise multiplication is highly inefficient in terms of both memory and computation. Since the loss is only non-zero at the target label indices, we can use advanced indexing to extract the target log-probabilities directly. This avoids allocating the large one-hot tensor and performing redundant operations.
loss_per_sample = -log_probs[jnp.arange(x.shape[0]), labels].astype(jnp.float32)
Adds
JAXBench/benchmark/level3/: 30 backward-pass (VJP) workloads built from 16 Tokamax ops, onebaseline.pyper workload.Nothing in this PR has run on a TPU. Everything below was checked on CPU with JAX 0.12.0 (Python 3.13); the Pallas kernels ran only in Pallas interpret mode. Two forward-kernel checks fail and one kernel could not be executed at all; see "Known issues".
What is added
Code is taken from openxla/tokamax at commit
b4b6eb35b4f05fc885f8582d360ff10e2cfc0625. Each file is self-contained (standard library,jax,numpyonly) and names the Tokamax files it was copied from, with their copyright lines.Two tracks per op. In both,
workloadreturns the gradients and is the correctness oracle.<N>p_<Op>_Bwd): a pure-JAX forward reference only. The task is to derive the backward pass and write the Pallas TPU kernels.<N>p_<Op>_Bwd_PallasFwd): the same file plus Tokamax's forward-only Pallas TPU kernel, inlined and exposed asforward_pallas. The task is to write the Pallas backward kernel that pairs with it.No file contains a Pallas backward kernel. With one exception (fused FP8 matmul, below), no file contains a hand-derived backward: the gradients are
jax.vjpof the forward reference.attention(dot-product attention)66p_Dot_Product_Attention_Bwd82p_Dot_Product_Attention_Bwd_PallasFwdsplash_attention67p_Splash_Attention_Bwd83p_Splash_Attention_Bwd_PallasFwdragged_dot68p_Ragged_Dot_Bwd84p_Ragged_Dot_Bwd_PallasFwdgmm_v269p_GMM_v2_Bwd85p_GMM_v2_Bwd_PallasFwdkda(Kimi Delta Attention)70p_KDA_Bwd86p_KDA_Bwd_PallasFwdlinear_softmax_cross_entropy_loss71p_Linear_Softmax_Cross_Entropy_Bwd87p_Linear_Softmax_Cross_Entropy_Bwd_PallasFwdnormalization(layer norm)72p_Layer_Norm_Bwdtriangle_multiplication73p_Triangle_Multiplication_Bwd88p_Triangle_Multiplication_Bwd_PallasFwdexperimental/fused_fp8_matmul74p_Fused_FP8_Matmul_Bwd89p_Fused_FP8_Matmul_Bwd_PallasFwdexperimental/gdn275p_GDN2_Bwd90p_GDN2_Bwd_PallasFwdexperimental/mhc_lite76p_MHC_Lite_Bwd91p_MHC_Lite_Bwd_PallasFwdexperimental/raven77p_Raven_Bwd92p_Raven_Bwd_PallasFwdexperimental/rwkv778p_RWKV7_Bwd93p_RWKV7_Bwd_PallasFwdgated_linear_unit79p_Gated_Linear_Unit_Bwdexperimental/moba80p_MoBA_Bwd94p_MoBA_Bwd_PallasFwdexperimental/nsa81p_NSA_Bwd95p_NSA_Bwd_PallasFwdFile interface
Same as level1/level2 (
CONFIG,create_inputs(dtype),workload(*inputs), optionalget_flops()), plus:forward(*primals): the pure-JAX forward reference.create_inputsreturns the arguments offorwardfollowed by the output cotangent.workload(*create_inputs())returns a tuple of gradients, one per entry ofVJP_SPEC['diff_argnums'].CONFIGSholds named configs, including a small'smoke'config for CPU.forward_pallas(*primals)returns(out, residuals), whereresidualsis what the Tokamax forward saves for its own backward.How it was checked (CPU)
Per workload, on the
'smoke'config unless noted:jit(workload)runs in float32 and bfloat16 and returns finite, non-zero gradients of the right shapes.workloadequalsjax.vjp(forward)recomputed independently (not applicable to fused FP8 matmul).forward_pallasin interpret mode is compared withforwardin bfloat16 at atol = rtol = 1e-2, and in float32 at 1e-3 where the kernel accepts float32.jit(workload)is recorded (compile only).Result: 28 of 30 workloads have no failing check; one of those (
85p) has both forward-kernel checks skipped because the kernel cannot run off TPU. For most ops the inlined kernel was also compared with the installed Tokamax kernel in interpret mode and is bit-identical, and for several ops the gradients were additionally compared with Tokamax's hand-written backward kernel or a float64 closed form.Known issues
Failing or unrun forward-kernel checks
83p_Splash_Attention_Bwd_PallasFwd: in bfloat16, 1 of 4.2 million output elements is outside 1e-2, at 1.004x the tolerance. The kernel rescalesqin the input dtype. The inlined kernel is bit-identical to the installed one.88p_Triangle_Multiplication_Bwd_PallasFwd: in bfloat16, 0.17% of output elements are outside 1e-2 on smoke (0.09% at the default config). The pure-JAX reference in bfloat16 is about as far from the float32 value as the kernel is. Tokamax's own tests for this kernel use much looser tolerances (rtol 0.05 with atol 0.15 or more), under which nothing is outside. The kernel accepts bfloat16 only. Bit-identical to the installed kernel.85p_GMM_v2_Bwd_PallasFwd: thegmm_v2kernel has never been executed. It cannot run in Pallas interpret mode in JAX 0.12 (dynamically sized row blocks and SMEM inputs); the installed kernel fails the same way. The inlined code differs from Tokamax in 5 lines (an absl flag replaced by its default) and traces to the same jaxpr as the library kernel. That shows the copy is faithful, not that the kernel matchesforward.91p_MHC_Lite_Bwd_PallasFwd: the kernel accepts bfloat16 only, and on CPU the interpreter's tiling always takes the multi-strip loop; the single-strip branch a TPU would take was lowered but never executed.Fused FP8 matmul does not use autodiff. The forward rounds to FP8, so
jax.vjp(forward)is not linear in the cotangent and is 16-23% away from the op's backward. Tokamax defines the backward by convention (straight-through, with the cotangent also quantised), so74pand89pcontain that pure-JAX reference backward as the oracle. These two files therefore give the solver a backward to translate rather than derive.Departures from the Tokamax references (each commented in the file):
78p,93p):forwardsums the per-chunk log-decay in float32, as the Pallas kernel does. Tokamax's reference sums in the input dtype, which put the bfloat16 oracle 3-6.6% away from the float32 gradients.67p,83p):forwardrepeats the key/value heads in float32, as the dot-product reference does. Otherwisedk/dvare rounded to bfloat16 per query head before being summed, and differ from Tokamax's own VJP.84p): the quantisation code is removed because it needsqwix.Opwrapper is replaced by a plainforward_pallasfunction with options fixed; pydantic configs become plain dataclasses and jaxtyping annotations are removed.Inputs that differ from the Tokamax tests
B * sqrt(H) / 4. With a cotangent of one every gradient entry on the smoke config is below 1e-2.N ** -0.5, ragged dot / GMM v2 0.1.dtypeis passed, following the Tokamax arg specs: layer-norm scale and offset, Raveng, KDAa_loganddelta_time_bias, mHC-liteh_post.Default configs smaller than the Tokamax arg specs. The pure-JAX autodiff oracle needs far more memory than the kernels do. Where XLA's plan (CPU compile, not a TPU measurement) was above about 6 GiB, the default is reduced and the arg-spec shape is kept as a second entry in
CONFIGS:Ragged dot (11.5 GiB) and GMM v2 (21.5 GiB) keep their arg-spec defaults: the CPU plan uses a dense loop over groups, whereas a TPU lowers
ragged_dot_generalto a native instruction, so the CPU figure may not apply.Harness: two things for reviewers
1. The correctness check does not suit gradients.
harness/correctness.pyusesnp.allclose(atol=1e-2, rtol=1e-2)per element, in bfloat16. For these workloads that has two failure modes, both measured:check_correctnessaccepted an all-zero gradient. Inputs and cotangents were scaled to avoid this, but some gradients remain mostly below the tolerance (for example KDAdg, about 4% of elements above 1e-2).workloadin bfloat16 differs by more than 1e-2 fromworkloadon the same values in float32, which is what a kernel that accumulates in float32 would be compared against:dx)dlhs), 62% (drhs) on smokedwon smoke, 15% at 128x128ds), 0.9% (dg) on smoke; 0.006%, 0.064% at the default configFor Raven, Tokamax's own Pallas backward kernel was run against the oracle with this check: it passes at the default config and fails on smoke. A scale-aware comparison (for example relative L2 per gradient) would address both failure modes. This PR does not change the harness.
2. The harness does not discover these workloads.
python -m JAXBench listreports 0 workloads:benchmark/__init__.pylooks forbenchmark/<name>/baseline.pyand does not descend intolevel1/,level2/orlevel3/. Not changed here.maxkernel/adk/evaluation/jaxbench_adapted_dataset/has nolevel3counterpart yet either.Not verified
workloadonly and notforward_pallas.get_flopsformulas were not compared with XLA cost analysis.