Safe Rust wrappers for cuTENSOR, NVIDIA's library for tensor contractions, reductions, permutations, and elementwise operations on the GPU.
An operation is described by TensorDescriptors and an OperationDescriptor, planned into a Plan, and run through methods on the plan.
This crate provides:
- Tensor descriptors: dense tensors with explicit extents, strides, and alignment, and block-sparse tensors.
- Operations: contractions of two or three tensors, reductions, permutations, elementwise binary and trinary operations, and block-sparse contractions, with modes labelled by characters.
- Planning: kernel-selection preferences, JIT-compiled kernels, workspace estimates, and plan and kernel caches that can be saved to files.
- Multi-GPU and multi-process: cuTENSORMg and cuTENSORMp, behind the
mgandmpfeatures.
Contract D[m, n] = A[m, k] * B[k, n], a matrix product written as a tensor contraction:
use singe_cuda::{context::Context as CudaContext, memory::DeviceMemory, stream::StreamOptions};
use singe_cutensor::{
context::Context,
operation::{ComputeDescriptor, OperationDescriptor, TensorOperand},
plan::{Plan, PlanPreference},
tensor::TensorDescriptor,
types::{Mode, WorkspacePreference},
};
fn main() -> anyhow::Result<()> {
let cuda = CudaContext::create()?;
let context = Context::create(&cuda)?;
let stream = cuda.create_stream(StreamOptions::default())?;
let modes_a = [Mode::from('m'), Mode::from('k')];
let modes_b = [Mode::from('k'), Mode::from('n')];
let modes_d = [Mode::from('m'), Mode::from('n')];
let descriptor_a = TensorDescriptor::create_for::<f32>(&context, &[2, 4], 128)?;
let descriptor_b = TensorDescriptor::create_for::<f32>(&context, &[4, 3], 128)?;
let descriptor_d = TensorDescriptor::create_for::<f32>(&context, &[2, 3], 128)?;
let contraction = OperationDescriptor::contraction(
&context,
TensorOperand::identity(&descriptor_a, &modes_a),
TensorOperand::identity(&descriptor_b, &modes_b),
TensorOperand::identity(&descriptor_d, &modes_d),
TensorOperand::identity(&descriptor_d, &modes_d),
ComputeDescriptor::f32(),
)?;
let preference = PlanPreference::create_default(&context)?;
let workspace_limit = Plan::estimate_workspace_size(
&context,
&contraction,
&preference,
WorkspacePreference::Default,
)?;
let plan = Plan::create(&context, &contraction, &preference, workspace_limit)?;
let mut workspace = DeviceMemory::<u8>::create_uninit(&cuda, plan.workspace_size())?;
let a = DeviceMemory::from_slice(&cuda, &[1.0_f32; 8])?;
let b = DeviceMemory::from_slice(&cuda, &[0.5_f32; 12])?;
let c = DeviceMemory::<f32>::zeroed(&cuda, 6)?;
let mut d = DeviceMemory::<f32>::zeroed(&cuda, 6)?;
// D = 1 * A * B + 0 * C
plan.contract(
&1.0_f32,
&a,
&b,
&0.0_f32,
&c,
&mut d,
Some(workspace.output()),
&stream,
)?;
assert_eq!(d.copy_to_host_vec()?, [2.0; 6]);
Ok(())
}Plan::contract checks every operand against the plan, rejects stream capture, and waits for the stream before it returns.
Each operation also has an unsafe *_async form that runs the same checks and only enqueues the work; its caller keeps the plan, operands, workspace, and scalars alive until the work finishes.
mg: themgmodule, for cuTENSORMg, which splits tensors across several GPUs in one process.mp: thempmodule, for cuTENSORMp, which distributes tensors across processes and communicates throughsinge-nccl.
- Contraction: plan and run
D = alpha * A * B + beta * Con four-mode tensors. - Trinary contraction: contract three tensors in one operation.
- JIT contraction: compile a kernel at plan time and reuse the kernel cache file.
- Plan cache: reuse contraction planning work through the plan cache file.
- Einsum: express contractions and reductions as einsum equations.
- Reduction: reduce tensor modes.
- Permutation: reorder tensor modes.
- Permutation with padding: permute into a padded output.
- Elementwise binary and elementwise trinary: combine two or three tensors.
- Block-sparse contraction: contract block-sparse tensors.
- Multi-GPU contraction: distribute a contraction over the listed GPUs, with
--features mg. - Multi-GPU blog post: the cuTENSORMg blog post's contraction over a power-of-two number of GPUs, with
--features mg. - Multi-process contraction: a cuTENSORMp contraction with one process per rank, configured through
SINGE_CUTENSOR_MP_*environment variables, with--features mp.