Choosing a Library
NVIDIA's math libraries are tuned per architecture by engineers with access to the SASS scheduler, the microarchitecture team, and hardware that hasn't shipped yet. As a rule of thumb rather than a measured benchmark — the actual gap varies by shape, precision, and architecture generation — a hand-written GEMM that reaches something like 60% of cuBLAS's throughput on the same shapes, on the same architecture, is realistically a good hand-written GEMM. The remaining gap is instruction scheduling, tile-size search, and register allocation tuned per compute capability by people who do nothing else. Programming Tensor Cores makes this same point about wmma kernels specifically; this page generalizes it to the whole library landscape and gives a rule for when to reach for one, and when not to.
cuBLAS
cuBLAS is NVIDIA's implementation of the BLAS (Basic Linear Algebra Subprograms) interface on the GPU: vector-vector, matrix-vector, and matrix-matrix operations, including the GEMM that Choosing a Library and Programming Tensor Cores both treat as the target hand-written kernels are measured against. The API is small and stable — a handle, a stream, and a handful of call shapes — but it inherits one convention from Fortran BLAS that trips up nearly everyone writing C or C++ against it for the first time.
cuDNN
cuDNN is NVIDIA's library of tuned primitives for deep learning why the first iteration on a new input shape is slow, why results can differ slightly between runs, and why changing a batch size can suddenly change performance by more than the batch size alone would predict.
Math Libraries
Beyond dense linear algebra and deep learning, four more CUDA libraries cover the numerical building blocks that show up constantly but rarely justify a hand-written kernel: Fourier transforms, random number generation, sparse linear algebra, and dense/sparse factorizations. Each has its own handle type and its own lifecycle, but — as the closing section here makes explicit — they share more structure with each other, and with cuBLAS, than the four separate APIs first suggest.
Thrust
Thrust is a C++ template library, shipped with the CUDA Toolkit, that reproduces the shape of the C++ Standard Template Library on the GPU the host-callable layer above cub's in-kernel primitives.
CUB
CUB is the layer Thrust is built on and Choosing a Library already named as the tuned building block for reductions, scans, and sorts: a template library of GPU primitives available at three different scopes, chosen depending on whether the call site is host code, a whole kernel, or a handful of cooperating threads inside one. Where Thrust replaces a kernel you'd otherwise write, CUB is what you reach for when you're still writing the kernel yourself and want a tuned, per-architecture primitive as one piece of it.
CUTLASS
CUTLASS is a C++ template library for building GEMM (and convolution) kernels with cuBLAS-class performance out of composable, reusable pieces — tile shapes, memory-movement stages, and epilogues — rather than a single pre-built call. cuBLAS is fast but fixed: cublasSgemm computes C = alpha op(A) op(B) + beta * C and nothing else, in the layouts and precisions it was built for. CUTLASS exists for the shapes and fusions that fall outside that fixed surface — an unusual data type, a custom epilogue, a problem size cuBLAS's kernel selection handles badly — while still generating code tuned close to cuBLAS's own throughput on the same hardware.
NCCL
NCCL (NVIDIA Collective Communications Library) is the library that moves data between GPUs — within a node over NVLink or PCIe, and across nodes over the network — through a small set of collective operations borrowed from the MPI world setting up a communicator, the collectives themselves, and how a call integrates with a stream. Collectives with NCCL covers the harder half — ring versus tree algorithm selection, and overlapping communication with gradient computation in a real training loop — and builds directly on the vocabulary defined here.
CUDA Python & CuPy
Two different Python packages both answer to "CUDA in Python," and confusing them is the first mistake most people make. NVIDIA's cuda-python is a thin, official binding to the driver and runtime APIs — the same calls this section has been making in C++, now callable from Python. CuPy is a third-party, NumPy-compatible array library built on top of those bindings. Almost nobody wants the first one directly; almost everybody wants the second.
Numba CUDA
CuPy covers array operations, and cp.RawKernel covers the case where you need a real kernel — but that kernel is a CUDA C++ string embedded in a Python file, with no syntax highlighting, no type checking, and no debugger. Numba takes the other route: you write the kernel in Python, decorated with @cuda.jit, and Numba compiles that Python function to PTX at first call.
PyTorch Extensions
PyTorch composes: nearly anything can be built from existing operators. What composition cannot always give you is one kernel. A sequence of PyTorch ops writes every intermediate tensor to global memory and reads it back for the next op, so a chain of cheap elementwise operations spends almost all its time moving data — the problem Kernel Fusion and Launch Overhead covers in general.
Triton
Writing a fused kernel in CUDA C++ means writing the fusion and everything around it: the thread-to-element mapping, the shared-memory staging, the vectorized loads, the bank-conflict-free layout. Most of that work is mechanical, most of it is where the bugs live, and none of it is the algorithm you actually wanted to express.