libsdca.jax provides classification and distillation losses as JAX functions that
compose with jit, grad and vmap. Install libsdca[jax] for CPU, or
libsdca[jax-cuda] for Linux NVIDIA GPUs (CUDA 13 runtime and Triton included;
Python 3.11+ and NVIDIA driver 580+):
import jax
from libsdca.jax import cross_entropy, topk_hinge_loss
def objective(w, x, y):
return cross_entropy(x @ w, y) # or topk_hinge_loss(x @ w, y, k=2)
step = jax.jit(jax.grad(objective))Every function takes reduction ("mean", "sum", "none"), axis=-1, backend
("auto", "jax", "native") and precision; the hyperparameters are keyword-only:
| Function | Loss | Targets | Parameters |
|---|---|---|---|
cross_entropy |
Softmax cross-entropy | labels or class probabilities |
weight, ignore_index=None, label_smoothing=0.0
|
topk_cross_entropy |
Top-k entropy | labels |
k (required), weight, ignore_index
|
multiclass_hinge_loss |
Crammer–Singer multiclass hinge | labels |
gamma=0.0, weight, ignore_index
|
topk_hinge_loss |
Top-k hinge, |
labels |
k (required), variant="beta", gamma=0.0, weight, ignore_index
|
proximal_cross_entropy |
Moreau envelope of cross-entropy | labels |
alpha=1.0, weight, ignore_index
|
multilabel_cross_entropy |
Cross-entropy to the uniform target on the mask | boolean mask | — |
multilabel_hinge_loss |
Multilabel hinge | boolean mask | gamma=0.0 |
distillation_loss |
Logit distillation | teacher logits | temperature=1.0 |
proximal_distillation_loss |
Its Moreau envelope | teacher logits |
temperature=1.0, alpha=1.0
|
Each loss takes one kind of target: labels are integers of any dtype with the shape of
logits without axis; masks are booleans with the shape of logits. A boolean target of
a single-label loss raises TypeError, as an integer target of a multilabel loss does. See
API.md and
parameter domains.
-
weight,ignore_index,label_smoothingfollowtorch.nn.functional.cross_entropy: each example's loss is multiplied by its label's weight, examples whose label equalsignore_indexadd nothing (no loss, no gradient), and the mean divides by the sum of the weights of the other labels. Label smoothing mixes$(1 - \varepsilon)$ of the label with$\varepsilon$ of the uniform distribution:$(1-\varepsilon),w_y \ell + \frac{\varepsilon}{C} \sum_j w_j (\operatorname{lse}(z) - z_j)$ . Probability targets give$-\sum_j w_j t_j \log \operatorname{softmax}(z)_j$ , their mean divides by the number of examples, and they take noignore_index. The calls without these options run unchanged; with them, the composition isjax.numpyaround the same per-example losses. -
Static hyperparameters:
k,gamma,alphaandtemperatureselect kernels and working precision, so they must be Python numbers (or concrete 0-d arrays); a traced value raisesTypeError(underjit, close over them or mark them static).weightandlabel_smoothingmay be traced. -
precision:None(the storage dtype's work; 16-bit logits work infloat32),"float32","float64", orjnp.float32/jnp.float64."float64"needsjax_enable_x64on every backend (ValueErrorotherwise).
| Loss |
backend="auto" selection |
Derivatives |
|---|---|---|
cross_entropy, multilabel_cross_entropy, topk_cross_entropy ( |
Shared Triton kernels on CUDA; "jax" on CPU/TPU |
Any order |
| The other losses of labels and masks | Shared Triton kernels on CUDA; "jax" on TPU and on CPU for calls of jax_enable_x64 or precision="float32", except proximal_cross_entropy); otherwise "native"
|
First order (reverse) |
distillation_loss, proximal_distillation_loss
|
Shared Triton kernels on CUDA; the native CPU kernels through the FFI handlers; else "jax"
|
See Distillation |
- CUDA (
"auto"): Runs the same fused Triton kernels aslibsdca.torch, as one XLA FFI call underjit,grad,vmapandjax.export(passdisabled_checks=[jax.export.DisabledSafetyCheck.custom_call("libsdca_cuda_kernels")]; the export holds kernels for the exporting GPU's architecture). Per-row results match PyTorch. Linux wheels include the handler; without Triton 3.3–3.9 the losses keepjax.numpy(SDCA_JAX_CUDA; with a one-time warning when Triton is missing or older). CPU-committed arguments stay on the CPU. "jax": Purejax.numpy/ XLA, with no host round trip and fulljax.exportsupport. Losses that default tofloat64work needjax_enable_x64=Truehere: without it,backend="auto"takes the native library andbackend="jax"raises."native": Calls the C++ library with a custom VJP on CPU threads. On CPU it compiles a zero-copy XLA FFI handler on first use (SDCA_JAX_FFI, no PyTorch needed), else falls back tojax.pure_callback. The distillation losses run natively only through the FFI handlers on the CPU (NotImplementedErrorelsewhere); probability targets have no native path (NotImplementedError).- Subnormals: XLA on CPU flushes subnormal inputs and outputs to zero, so CPU losses and
gradient entries below
exp(-87.3)(float32) orexp(-708.4)(float64) are0. Sums of many individually underflowing exponentials are still kept (float32cross-entropy over 1,000 classes 90 below the target returns8.19e-37).
JAX 0.4.35 is supported with jit, grad and vmap; the shared native kernels need a JAX
that exposes jax.ffi. See environment variables for
the switches above.
On a multithreaded CPU, the array paths of cross_entropy, multilabel_cross_entropy,
topk_cross_entropy (SDCA_JAX_CPU_FUSION selects the variant:
auto(default): the first trace of a loss times both variants once at 4096 × 1024 (proximal distillation follows a fixed rule instead), keeps the fenced one unless the unfenced one takes at most 0.85 of its time, and caches the decision inSDCA_CACHE_DIRper CPU, thread setting, dtype and library version.fencedorunfenced: forces one variant.
The variants differ only in summation order. The setting is read when a loss is traced; call
jax.clear_caches() after changing it.
- Wrong types (a non-floating
logitsarray, a target of the wrong kind, a non-stringreduction, a non-integerk, a tracedgamma, ...) raiseTypeError; bad values (shapes,koutside$[1, C-1]$ , a negativegamma, an unknownreductionorbackend) raiseValueError, before any computation. - Data values: concrete (untraced) calls check them eagerly and raise
ValueErrorfor nonfinite logits, labels out of range or masks without a positive (or, formultilabel_hinge_loss, a negative) label. Insidejitorvmap, such rows givenanlosses and gradients on every backend, as do non-convergentproximal_cross_entropyrows: after a failed native call the native backend repeats the call row by row, so valid calls cost nothing extra.
distillation_loss and proximal_distillation_loss take student and teacher logits (a
different teacher dtype is promoted with the student's); the teacher receives no gradient and
equal logits give an exact zero gradient. No jax.hessian; proximal distillation supports
jax.hessian (forward-over-reverse) only.
python -m pytest -n auto tests/python --group jax # from a source checkout