Skip to content

JAXBench: add level3 backward-pass workloads from Tokamax - #146

Open
cccathygao wants to merge 1 commit into
mainfrom
cathygao-jaxbench-level3
Open

cccathygao wants to merge 1 commit into
mainfrom
cathygao-jaxbench-level3

Conversation

@cccathygao

Copy link
Copy Markdown
Collaborator

Adds JAXBench/benchmark/level3/: 30 backward-pass (VJP) workloads built from 16 Tokamax ops, one baseline.py per workload.

Nothing in this PR has run on a TPU. Everything below was checked on CPU with JAX 0.12.0 (Python 3.13); the Pallas kernels ran only in Pallas interpret mode. Two forward-kernel checks fail and one kernel could not be executed at all; see "Known issues".

What is added

Code is taken from openxla/tokamax at commit b4b6eb35b4f05fc885f8582d360ff10e2cfc0625. Each file is self-contained (standard library, jax, numpy only) and names the Tokamax files it was copied from, with their copyright lines.

Two tracks per op. In both, workload returns the gradients and is the correctness oracle.

  • Track 1 (<N>p_<Op>_Bwd): a pure-JAX forward reference only. The task is to derive the backward pass and write the Pallas TPU kernels.
  • Track 2 (<N>p_<Op>_Bwd_PallasFwd): the same file plus Tokamax's forward-only Pallas TPU kernel, inlined and exposed as forward_pallas. The task is to write the Pallas backward kernel that pairs with it.

No file contains a Pallas backward kernel. With one exception (fused FP8 matmul, below), no file contains a hand-derived backward: the gradients are jax.vjp of the forward reference.

Tokamax op Track 1 Track 2
attention (dot-product attention) 66p_Dot_Product_Attention_Bwd 82p_Dot_Product_Attention_Bwd_PallasFwd
splash_attention 67p_Splash_Attention_Bwd 83p_Splash_Attention_Bwd_PallasFwd
ragged_dot 68p_Ragged_Dot_Bwd 84p_Ragged_Dot_Bwd_PallasFwd
gmm_v2 69p_GMM_v2_Bwd 85p_GMM_v2_Bwd_PallasFwd
kda (Kimi Delta Attention) 70p_KDA_Bwd 86p_KDA_Bwd_PallasFwd
linear_softmax_cross_entropy_loss 71p_Linear_Softmax_Cross_Entropy_Bwd 87p_Linear_Softmax_Cross_Entropy_Bwd_PallasFwd
normalization (layer norm) 72p_Layer_Norm_Bwd none (Tokamax has no TPU kernel)
triangle_multiplication 73p_Triangle_Multiplication_Bwd 88p_Triangle_Multiplication_Bwd_PallasFwd
experimental/fused_fp8_matmul 74p_Fused_FP8_Matmul_Bwd 89p_Fused_FP8_Matmul_Bwd_PallasFwd
experimental/gdn2 75p_GDN2_Bwd 90p_GDN2_Bwd_PallasFwd
experimental/mhc_lite 76p_MHC_Lite_Bwd 91p_MHC_Lite_Bwd_PallasFwd
experimental/raven 77p_Raven_Bwd 92p_Raven_Bwd_PallasFwd
experimental/rwkv7 78p_RWKV7_Bwd 93p_RWKV7_Bwd_PallasFwd
gated_linear_unit 79p_Gated_Linear_Unit_Bwd none (Tokamax has no TPU kernel)
experimental/moba 80p_MoBA_Bwd 94p_MoBA_Bwd_PallasFwd
experimental/nsa 81p_NSA_Bwd 95p_NSA_Bwd_PallasFwd

File interface

Same as level1/level2 (CONFIG, create_inputs(dtype), workload(*inputs), optional get_flops()), plus:

  • forward(*primals): the pure-JAX forward reference.
  • create_inputs returns the arguments of forward followed by the output cotangent.
  • workload(*create_inputs()) returns a tuple of gradients, one per entry of VJP_SPEC['diff_argnums'].
  • CONFIGS holds named configs, including a small 'smoke' config for CPU.
  • Track 2: forward_pallas(*primals) returns (out, residuals), where residuals is what the Tokamax forward saves for its own backward.

How it was checked (CPU)

Per workload, on the 'smoke' config unless noted:

  1. jit(workload) runs in float32 and bfloat16 and returns finite, non-zero gradients of the right shapes.
  2. workload equals jax.vjp(forward) recomputed independently (not applicable to fused FP8 matmul).
  3. The gradients match the installed Tokamax library's own op and VJP in float32 (atol = rtol = 1e-4).
  4. Track 2: forward_pallas in interpret mode is compared with forward in bfloat16 at atol = rtol = 1e-2, and in float32 at 1e-3 where the kernel accepts float32.
  5. The default config is shape-traced, and XLA's memory plan for jit(workload) is recorded (compile only).

Result: 28 of 30 workloads have no failing check; one of those (85p) has both forward-kernel checks skipped because the kernel cannot run off TPU. For most ops the inlined kernel was also compared with the installed Tokamax kernel in interpret mode and is bit-identical, and for several ops the gradients were additionally compared with Tokamax's hand-written backward kernel or a float64 closed form.

Known issues

Failing or unrun forward-kernel checks

  • 83p_Splash_Attention_Bwd_PallasFwd: in bfloat16, 1 of 4.2 million output elements is outside 1e-2, at 1.004x the tolerance. The kernel rescales q in the input dtype. The inlined kernel is bit-identical to the installed one.
  • 88p_Triangle_Multiplication_Bwd_PallasFwd: in bfloat16, 0.17% of output elements are outside 1e-2 on smoke (0.09% at the default config). The pure-JAX reference in bfloat16 is about as far from the float32 value as the kernel is. Tokamax's own tests for this kernel use much looser tolerances (rtol 0.05 with atol 0.15 or more), under which nothing is outside. The kernel accepts bfloat16 only. Bit-identical to the installed kernel.
  • 85p_GMM_v2_Bwd_PallasFwd: the gmm_v2 kernel has never been executed. It cannot run in Pallas interpret mode in JAX 0.12 (dynamically sized row blocks and SMEM inputs); the installed kernel fails the same way. The inlined code differs from Tokamax in 5 lines (an absl flag replaced by its default) and traces to the same jaxpr as the library kernel. That shows the copy is faithful, not that the kernel matches forward.
  • 91p_MHC_Lite_Bwd_PallasFwd: the kernel accepts bfloat16 only, and on CPU the interpreter's tiling always takes the multi-strip loop; the single-strip branch a TPU would take was lowered but never executed.

Fused FP8 matmul does not use autodiff. The forward rounds to FP8, so jax.vjp(forward) is not linear in the cotangent and is 16-23% away from the op's backward. Tokamax defines the backward by convention (straight-through, with the cotangent also quantised), so 74p and 89p contain that pure-JAX reference backward as the oracle. These two files therefore give the solver a backward to translate rather than derive.

Departures from the Tokamax references (each commented in the file):

  • RWKV-7 (78p, 93p): forward sums the per-chunk log-decay in float32, as the Pallas kernel does. Tokamax's reference sums in the input dtype, which put the bfloat16 oracle 3-6.6% away from the float32 gradients.
  • Splash attention (67p, 83p): forward repeats the key/value heads in float32, as the dot-product reference does. Otherwise dk/dv are rounded to bfloat16 per query head before being summed, and differ from Tokamax's own VJP.
  • Ragged dot v1 kernel (84p): the quantisation code is removed because it needs qwix.
  • In every Track 2 file the Tokamax Op wrapper is replaced by a plain forward_pallas function with options fixed; pydantic configs become plain dataclasses and jaxtyping annotations are removed.

Inputs that differ from the Tokamax tests

  • MoBA and NSA: keys and queries carry block structure so top-k block routing is well separated. With unstructured random inputs the selection depended on rounding (at T=8192 in bfloat16 the MoBA kernel and its reference picked different blocks in 1 of the 512 query blocks of one head).
  • Cross-entropy loss: weights scaled to fan-in, and an explicit scalar cotangent B * sqrt(H) / 4. With a cotangent of one every gradient entry on the smoke config is below 1e-2.
  • GLU: weights scaled to fan-in. Triangle multiplication: Tokamax's fan-in-scaled test inputs.
  • Cotangent scales chosen so gradients are of order 0.05-3: Raven 0.5, RWKV-7 0.25, NSA 2, triangle multiplication N ** -0.5, ragged dot / GMM v2 0.1.
  • Some parameter inputs stay float32 whatever dtype is passed, following the Tokamax arg specs: layer-norm scale and offset, Raven g, KDA a_log and delta_time_bias, mHC-lite h_post.

Default configs smaller than the Tokamax arg specs. The pure-JAX autodiff oracle needs far more memory than the kernels do. Where XLA's plan (CPU compile, not a TPU measurement) was above about 6 GiB, the default is reduced and the arg-spec shape is kept as a second entry in CONFIGS:

Op Default Planned temporaries Arg-spec shape
dot-product attention B1, T4096, 16 heads 5.09 GiB 326 GiB
splash attention 16 heads, T4096 5.09 GiB 80.8 GiB
KDA 4 heads, T4096 3.92 GiB 52.2 GiB
GDN-2 8 heads, T2048 3.04 GiB about 12 GiB
RWKV-7 8 heads, T2048 5.88 GiB 23.5 GiB
mHC-lite T8192 0.75 GiB 18.0 GiB
NSA B2, T4096 3.30 GiB 13.9 GiB

Ragged dot (11.5 GiB) and GMM v2 (21.5 GiB) keep their arg-spec defaults: the CPU plan uses a dense loop over groups, whereas a TPU lowers ragged_dot_general to a native instruction, so the CPU figure may not apply.

Harness: two things for reviewers

1. The correctness check does not suit gradients. harness/correctness.py uses np.allclose(atol=1e-2, rtol=1e-2) per element, in bfloat16. For these workloads that has two failure modes, both measured:

  • Too lenient for small values. Before rescaling, every cross-entropy gradient entry was below 1e-2 and check_correctness accepted an all-zero gradient. Inputs and cotangents were scaled to avoid this, but some gradients remain mostly below the tolerance (for example KDA dg, about 4% of elements above 1e-2).
  • Too strict where the reference itself rounds. The table gives the fraction of gradient elements where workload in bfloat16 differs by more than 1e-2 from workload on the same values in float32, which is what a kernel that accumulates in float32 would be compared against:
Workload Fraction outside 1e-2
Triangle multiplication 22-67% of the weight gradients on smoke (0% for dx)
Fused FP8 matmul 18% (dlhs), 62% (drhs) on smoke
GLU 6% of dw on smoke, 15% at 128x128
mHC-lite 0.02% of one gradient on smoke; 5% of the pre-gate weight gradient at the default config
Raven 0.3% (ds), 0.9% (dg) on smoke; 0.006%, 0.064% at the default config
All other ops 0% on smoke

For Raven, Tokamax's own Pallas backward kernel was run against the oracle with this check: it passes at the default config and fails on smoke. A scale-aware comparison (for example relative L2 per gradient) would address both failure modes. This PR does not change the harness.

2. The harness does not discover these workloads. python -m JAXBench list reports 0 workloads: benchmark/__init__.py looks for benchmark/<name>/baseline.py and does not descend into level1/, level2/ or level3/. Not changed here. maxkernel/adk/evaluation/jaxbench_adapted_dataset/ has no level3 counterpart yet either.

Not verified

  • Anything on a TPU: Mosaic compilation, VMEM fit, the block-size heuristics on real devices, timing, and whether bfloat16 on TPU reproduces the CPU oracle values.
  • Default configs that were only shape-traced, never run numerically: ragged dot, GMM v2, GDN-2, RWKV-7, GLU, fused FP8 matmul. The others were run at least once on CPU, in some cases workload only and not forward_pallas.
  • get_flops formulas were not compared with XLA cost analysis.
  • JAX versions other than 0.12.0.

30 workloads under JAXBench/benchmark/level3, one baseline.py each, built
from 16 Tokamax ops (openxla/tokamax @ b4b6eb3):

- Track 1 (16 workloads): pure-JAX forward reference; workload() returns the
  gradients via jax.vjp.
- Track 2 (14 workloads): the same plus Tokamax's forward-only Pallas TPU
  kernel, inlined as forward_pallas().

Checked on CPU only (Pallas interpret mode); nothing has run on a TPU.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces pure-JAX reference implementations and benchmark harnesses for several JAXBench level 3 backward pass operators, covering various attention mechanisms, normalization layers, and matrix multiplications. The feedback highlights critical performance and correctness issues in the added baselines. Specifically, in the KDA operators, the nested jax.lax.fori_loop should be refactored using jax.vmap and jax.lax.scan to allow parallelization and support reverse-mode automatic differentiation (jax.vjp). Additionally, in the linear softmax cross-entropy operators, the memory-intensive creation of a full one-hot tensor should be replaced with advanced indexing to directly extract target log-probabilities.

Comment on lines +120 to +153
states = jnp.zeros((batch, 1, heads, key_dim, value_dim), dtype=acc_dtype)
output_h = jnp.zeros((heads, batch, seq_len, value_dim), dtype=acc_dtype)

def step_token(h, b, t, carry):
states, output_h = carry
previous_state = states[b, 0, h]
state = previous_state * jnp.exp(g_h[h, b, t])[:, None]
prediction = k_h[h, b, t] @ state
residual = v_h[h, b, t] - prediction
new_state = state + (
beta_h[h, b, t] * k_h[h, b, t]
)[:, None] * residual[None, :]
out_t = q_h[h, b, t] @ new_state
output_h = output_h.at[h, b, t].set(out_t)
states = states.at[b, 0, h].set(new_state)
return states, output_h

def step_head(h, carry):
states, output_h = carry

def body_b(b, b_carry):
def body_t(t, t_carry):
return step_token(h, b, t, t_carry)

return jax.lax.fori_loop(0, seq_len, body_t, b_carry)

states, output_h = jax.lax.fori_loop(
0, batch, body_b, (states, output_h)
)
return states, output_h

states, output_h = jax.lax.fori_loop(
0, heads, step_head, (states, output_h)
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Using nested jax.lax.fori_loop to sequentially iterate over heads and batch elements is highly inefficient on accelerators (TPU/GPU) as it prevents parallelization. Furthermore, fori_loop is implemented using while_loop under the hood in JAX, which does not support reverse-mode automatic differentiation (jax.vjp), causing workload to fail at runtime.\n\nWe can rewrite this using jax.vmap to parallelize over heads and batch dimensions, and jax.lax.scan to efficiently propagate the recurrent state over the sequence dimension. This is both fully differentiable and highly performant.

  def step_fn(state, inputs):\n    q_t, k_t, v_t, g_t, beta_t = inputs\n    state = state * jnp.exp(g_t)[:, None]\n    prediction = k_t @ state\n    residual = v_t - prediction\n    state = state + (beta_t * k_t)[:, None] * residual[None, :]\n    out_t = q_t @ state\n    return state, out_t\n\n  def scan_fn(q_hb, k_hb, v_hb, g_hb, beta_hb):\n    s0 = jnp.zeros((key_dim, value_dim), dtype=acc_dtype)\n    _, out_hb = jax.lax.scan(step_fn, s0, (q_hb, k_hb, v_hb, g_hb, beta_hb))\n    return out_hb\n\n  vmap_fn = jax.vmap(jax.vmap(scan_fn, in_axes=0), in_axes=0)\n  output_h = vmap_fn(q_h, k_h, v_h, g_h, beta_h)

Comment on lines +120 to +153
"""KDA forward pass: explicit token-by-token delta-rule recurrence.

Args:
q: Queries, `[H, B, T, K]`. L2-normalised over `K` inside the op.
k: Keys, `[H, B, T, K]`. L2-normalised over `K` inside the op.
v: Values, `[H, B, T, V]`.
g: Raw per-channel gate input, `[H, B, T, K]`. The log-decay is
`-5 * sigmoid(exp(a_log) * (g + delta_time_bias))`.
beta: Per-token delta-rule learning rate, `[H, B, T]`, values in `[0, 1]`.
a_log: Per-head log decay rate, `[H]`.
delta_time_bias: Per-head, per-key-channel gate bias, `[H * K]`.

Returns:
The output, `[H, B, T, V]`, in the dtype of `q`. The state starts at zero
and the final state is not returned. Queries are scaled by `K ** -0.5`.
"""
heads, batch, seq_len, key_dim = q.shape
value_dim = v.shape[-1]
acc_dtype = _accumulator_dtype(q.dtype)
output_dtype = q.dtype
scale = key_dim**-0.5

g = _activate_gate(
g, a_log=a_log, delta_time_bias=delta_time_bias, lower_bound=_LOWER_BOUND
)
q_h = _l2_normalize(q, acc_dtype)
k_h = _l2_normalize(k, acc_dtype)

q_h = q_h * scale
v_h = v.astype(acc_dtype)
g_h = g.astype(acc_dtype)
beta_h = beta.astype(acc_dtype)

# One recurrent state per batch element and head.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Using nested jax.lax.fori_loop to sequentially iterate over heads and batch elements is highly inefficient on accelerators (TPU/GPU) as it prevents parallelization. Furthermore, fori_loop is implemented using while_loop under the hood in JAX, which does not support reverse-mode automatic differentiation (jax.vjp), causing workload to fail at runtime.\n\nWe can rewrite this using jax.vmap to parallelize over heads and batch dimensions, and jax.lax.scan to efficiently propagate the recurrent state over the sequence dimension. This is both fully differentiable and highly performant.

  def step_fn(state, inputs):\n    q_t, k_t, v_t, g_t, beta_t = inputs\n    state = state * jnp.exp(g_t)[:, None]\n    prediction = k_t @ state\n    residual = v_t - prediction\n    state = state + (beta_t * k_t)[:, None] * residual[None, :]\n    out_t = q_t @ state\n    return state, out_t\n\n  def scan_fn(q_hb, k_hb, v_hb, g_hb, beta_hb):\n    s0 = jnp.zeros((key_dim, value_dim), dtype=acc_dtype)\n    _, out_hb = jax.lax.scan(step_fn, s0, (q_hb, k_hb, v_hb, g_hb, beta_hb))\n    return out_hb\n\n  vmap_fn = jax.vmap(jax.vmap(scan_fn, in_axes=0), in_axes=0)\n  output_h = vmap_fn(q_h, k_h, v_h, g_h, beta_h)

Comment on lines +74 to +76
labels_one_hot = jax.nn.one_hot(labels, num_classes=w.shape[1], dtype=x.dtype)
loss = -labels_one_hot * log_probs
loss_per_sample = jnp.sum(loss, axis=-1, dtype=jnp.float32)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Creating a full one-hot tensor of shape (B, V) (where V is the vocabulary size, e.g., 128,256) and performing element-wise multiplication is highly inefficient in terms of both memory and computation. Since the loss is only non-zero at the target label indices, we can use advanced indexing to extract the target log-probabilities directly. This avoids allocating the large one-hot tensor and performing redundant operations.

Suggested change
labels_one_hot = jax.nn.one_hot(labels, num_classes=w.shape[1], dtype=x.dtype)
loss = -labels_one_hot * log_probs
loss_per_sample = jnp.sum(loss, axis=-1, dtype=jnp.float32)
loss_per_sample = -log_probs[jnp.arange(x.shape[0]), labels].astype(jnp.float32)

Comment on lines +74 to +76

# ==============================================================================
# Forward reference

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Creating a full one-hot tensor of shape (B, V) (where V is the vocabulary size, e.g., 128,256) and performing element-wise multiplication is highly inefficient in terms of both memory and computation. Since the loss is only non-zero at the target label indices, we can use advanced indexing to extract the target log-probabilities directly. This avoids allocating the large one-hot tensor and performing redundant operations.

  loss_per_sample = -log_probs[jnp.arange(x.shape[0]), labels].astype(jnp.float32)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant