libsdca.torch provides the classification losses as differentiable PyTorch functions and
modules (pip install 'libsdca[torch]'), on CPU and CUDA tensors in float16, bfloat16,
float32 and float64:
| Loss | Module | Function | Own options | Target |
|---|---|---|---|---|
| Cross-entropy | CrossEntropyLoss |
cross_entropy |
weight, ignore_index=-100, label_smoothing=0.0 |
labels or probabilities, classes on dim=1 |
| Linear + cross-entropy | LinearCrossEntropyLoss |
linear_cross_entropy |
class_weight, ignore_index=-100, label_smoothing=0.0, chunk_bytes, recompute |
labels or probabilities |
| Top-k cross-entropy | TopKCrossEntropyLoss |
topk_cross_entropy |
k (required, 1 <= k < C) |
labels |
| Multiclass hinge | MulticlassHingeLoss |
multiclass_hinge_loss |
gamma=0.0 (> 0 smooths) |
labels |
| Top-k hinge | TopKHingeLoss |
topk_hinge_loss |
k (required), variant="beta" (or "alpha"), gamma=0.0 |
labels |
| Proximal cross-entropy | ProximalCrossEntropyLoss |
proximal_cross_entropy |
alpha=1.0 (Moreau envelope) |
labels |
| Multilabel cross-entropy | MultilabelCrossEntropyLoss |
multilabel_cross_entropy |
boolean mask | |
| Multilabel hinge | MultilabelHingeLoss |
multilabel_hinge_loss |
gamma=0.0 |
boolean mask |
| Logit distillation | DistillationLoss |
distillation_loss |
temperature=1.0 |
teacher logits |
| Proximal distillation | ProximalDistillationLoss |
proximal_distillation_loss |
temperature=1.0, alpha=1.0 |
teacher logits |
Every loss also takes reduction="mean" ("none", "sum"), backend="auto" and
precision=None, and all but the cross-entropies the class dimension dim=-1. The
single-label losses besides cross-entropy take weight=None and ignore_index=None too.
Hyperparameters are keyword-only (modules mirror torch.nn: CrossEntropyLoss(weight)).
Labels are class indices of any integer dtype; the multilabel losses take boolean masks of
the logits' shape, and a target of the other kind raises TypeError.
import torch
from libsdca.torch import CrossEntropyLoss, TopKHingeLoss
logits = torch.randn(16, 10, requires_grad=True)
labels = torch.randint(0, 10, (16,))
CrossEntropyLoss()(logits, labels).backward()
TopKHingeLoss(k=2)(logits, labels).backward()On first use, libsdca compiles small C++ extensions against the installed PyTorch for its
fastest CPU and CUDA paths. This needs a C++17 compiler with OpenMP (build-essential on
Debian/Ubuntu, Xcode Command Line Tools on macOS), Python headers (python3-dev; included
in conda and uv-managed Pythons) and, on GPU, the CUDA headers. libsdca[torch] installs
ninja and, on Linux, a standalone Zig C compiler that Triton 3.4+ uses when the host has
none. Without build tools every loss still runs, on the prebuilt native library (CPU) or
Triton (CUDA), after a one-time warning; SDCA_CPU_EXTENSION=off and
SDCA_CUDA_LAUNCHER=off silence it (environment variables).
CrossEntropyLoss and cross_entropy take the arguments of torch.nn.CrossEntropyLoss and
torch.nn.functional.cross_entropy:
from libsdca.torch import CrossEntropyLoss
criterion = CrossEntropyLoss() # drop-in for torch.nn.CrossEntropyLoss()
loss = criterion(net(features), target) # raw logits, class labelsWhen a prediction is confident, naive float32 and float64 for two classes). CrossEntropyLoss keeps both down
to the output dtype's subnormal range (float32, float64):
logits = torch.tensor([[0., -20.]], requires_grad=True)
loss = CrossEntropyLoss()(logits, torch.tensor([0]))
loss.backward()
print(loss.item(), logits.grad) # 2.061e-9, [[-2.061e-9, 2.061e-9]]nn.CrossEntropyLoss feature |
libsdca.torch.CrossEntropyLoss |
|---|---|
weight, ignore_index=-100, reduction |
As PyTorch, including the weighted mean (float32 weights accepted for 16-bit autocast). |
label_smoothing |
As PyTorch: (1 - ε) times the loss plus ε / C times the smoothing term, which a logsumexp in tensor operations adds to the fused loss. |
| Probability targets | A floating-point target of the logits' shape, as PyTorch (log-softmax in tensor operations; ignore_index must stay negative). |
| Shapes and dtypes | (C,), (N, C), (N, C, d1, ...); non-contiguous inputs; labels of any integer dtype; 16-bit logits return a float32 loss and 16-bit gradients. |
| Invalid labels / nonfinite logits | ValueError on the CPU; NaN on CUDA (no host synchronization) and inside torch.func transforms. |
torch.compile, CUDA graphs, double backward |
Supported (compilation). |
Integer labels without smoothing take the fused kernels; smoothing and probability targets
work in float32 for 16- and 32-bit logits (float64 for float64 logits or
precision="float64"). The other losses place classes on dim=-1 (pass dim=1 for
(N, C, H, W)).
topk_cross_entropy, the multiclass and top-k hinges and proximal_cross_entropy take
weight ((C,), finite and nonnegative) and ignore_index (default None: no label is
ignored) with PyTorch's cross-entropy rule: each example's loss is multiplied by the weight
of its label, ignored examples contribute nothing (their logits are not read), and the mean
divides by the sum of the label weights. Without either option the call is the fused
kernel's.
linear_cross_entropy(hidden, weight, target, bias=None, *, class_weight=None, ignore_index=-100, reduction="mean", label_smoothing=0.0, backend="auto", precision=None, chunk_bytes=None, recompute=False) and LinearCrossEntropyLoss evaluate
cross_entropy(F.linear(hidden, weight, bias), target) in row chunks without materializing
the (N, V) logits:
from libsdca.torch import linear_cross_entropy
hidden = model.body(tokens) # (B, T, H)
loss = linear_cross_entropy(hidden, model.lm_head.weight, labels) # labels (B, T); -100 ignored
loss.backward()- Chunking and memory: Rows stream in chunks of
chunk_bytes(default:256 MiBof logits or2048rows, whichever is more; at least8192rows on CUDA BF16). For"mean"/"sum"on BF16/FP32/FP64, forward accumulatesgrad_hidden,grad_weightandgrad_biasinfloat32(float64forfloat64) at upstream derivative1(3 GEMMs total; peak extra memory isgrad_hidden, the parameter gradients and one chunk). Withrecompute=True,reduction="none", orfloat16, backward recomputes each chunk's logits (4 GEMMs total; nothing saved between forward and backward). - Label smoothing composes the per-example losses with the target logits and one matrix-vector product, still without the logits. Probability targets have the logits' size, and compute the logits in full.
- Precision and
GradScaler: Passrecompute=Truewhen using aGradScalerthat must recover subnormal gradient tails that would underflowfloat32at unit scale during forward. - Invalid rows: On CUDA, any unignored row with an out-of-range target or nonfinite
logits makes a
"mean"or"sum"loss and all reduced parameter gradientsNaN.
backend="auto" (default) selects the fastest path for each call; explain_backend(loss, *args, **kwargs) reports the path of a call, a function or module with its arguments:
from libsdca.torch import explain_backend, topk_hinge_loss
explain_backend(topk_hinge_loss, logits, labels, k=2)
# ('kernel', "the CPU extension's top-k kernels (float64)")Path (backend that selects it) |
Where backend="auto" takes it |
|---|---|
"kernel": fused Triton kernels (launched from C++ autograd or by Triton) or the CPU extension |
CUDA ( |
"native": the prebuilt native library on PyTorch's OpenMP pool (CPU tensors) |
CPU without build tools; float64 cross-entropy rows of more than 32 classes |
"torch": composed tensor operations (CPU or CUDA) |
Other CUDA shapes |
backend="kernel" behaves as "auto" but raises NotImplementedError where no fused
kernel computes the call.
Loss values have dtype promote(logits.dtype, float32); gradients keep logits.dtype.
precision takes None, "float32" or "float64" (or torch.float32, torch.float64).
precision |
Working arithmetic |
|---|---|
None (default), "float32" |
float32 work for 16- and 32-bit logits, with float64 row statistics and promotion of rows float32 cannot hold (API.md); float64 for float64 logits ("float32" raises) |
"float64" |
float64 throughout, rounded once to the output dtype |
16-bit logits halve memory traffic; arithmetic runs in float32 or float64
(known limits).
Wrong types raise TypeError (a non-tensor, integer logits, a boolean target for a
single-label loss, k=2.0); bad values, shapes and options raise ValueError (k=0,
precision="fp32" or torch.float16), in constructors as in calls. Data values are checked where it costs no synchronization: on the CPU nonfinite
logits, labels out of range and invalid class weights raise ValueError; on CUDA and inside
torch.func transforms the affected examples (or a reduction) are NaN.
On CUDA (the C++ launcher), backward for cross_entropy and the hinge, top-k and
multilabel losses (not distillation or the proximal losses) overwrites the logits tensor in
place when no other reference holds it (loss = criterion(net(x), y) or del logits before
loss.backward(); PyTorch 2.10 or later). On CPU, peak extra memory is one gradient-sized
tensor.
All CPU and CUDA losses are registered as torch.library custom ops and trace through
torch.compile(..., fullgraph=True) without graph breaks (from PyTorch 2.4; tested from 2.5).
Inside a compiled graph, Inductor does not overwrite the logits in place; to keep in-place
backward memory savings while compiling the model, wrap the loss with torch.compiler.disable.
torch.func transforms (grad, vjp, vmap, jacrev) support every loss: each runs its
eager path on the rows (a vmap adds the mapped dimension to the rows) and saves the
gradient. cross_entropy and CrossEntropyLoss also support forward mode and hessian;
the other losses are first order, and differentiating their gradient raises RuntimeError.
Fused kernels operate on full class rows (compatible with DDP, FSDP and pipeline
parallelism). Class-sharded DTensor inputs are supported via an all-gather fallback.
from libsdca.torch import DistillationLoss
criterion = DistillationLoss()
criterion(student_logits, teacher_logits).backward()Both inputs are raw logits; the teacher receives no gradient, no
Forward and backward on an NVIDIA A100 with PyTorch 2.15, against PyTorch implementations of
the same losses (F.cross_entropy for cross-entropy), eager and compiled alone with
torch.compile; median times in ms:
| Loss | Logits | dtype | libsdca |
torch.compile |
Eager PyTorch |
|---|---|---|---|---|---|
| Cross-entropy | 8192 × 128256 | bfloat16 |
4.73 | 4.97 | 13.55 |
| Cross-entropy | 8192 × 128256 | float32 |
9.09 | 9.33 | 27.24 |
| Multiclass hinge | 4096 × 32000 | bfloat16 |
0.53 | 1.99 | 6.52 |
| Multiclass hinge | 4096 × 32000 | float32 |
0.78 | 3.66 | 9.33 |
| Smoothed multiclass hinge | 4096 × 32000 | float32 |
2.88 | 24.34 | 41.92 |
The bfloat16 multiclass hinge needs 250 MiB of extra memory at its peak, against 750 MiB
compiled and 1,375 MiB eager.
python -m pytest -n auto tests/python --group torch # from a source checkout