Multi-GPU Basics
A workload that outgrows one GPU's memory or compute budget needs a second one, and every choice from there — how many processes, how work gets split, where the time actually goes — follows from a small set of rules about how CUDA treats "current device" as thread-local state. Get the discipline wrong and the symptom is rarely a crash; it's silent misallocation onto the wrong device or serialized work that looks like it should overlap.
Peer-to-Peer & NVLink
Two GPUs in the same box can talk to each other directly, or every byte between them can detour through host memory — the difference is entirely a matter of whether peer access has been enabled, and it has real bandwidth consequences either way. This page covers the API for checking, enabling, and using that direct path; the bandwidths themselves live on Interconnects: PCIe and NVLink.
Collectives with NCCL
The NCCL page covers the API — communicators, the collective calls, stream integration, grouped calls. This page covers what happens underneath an ncclAllReduce call: the ring and tree algorithms NCCL chooses between, the cost model that explains why ring all-reduce scales the way it does, and how a training loop overlaps communication with compute instead of paying for it serially. It doesn't repeat the API surface — link there for ncclCommInitRank, the collectives table, or ncclGroupStart/ncclGroupEnd.
Parallelism Strategies
Splitting a training job across GPUs means choosing what gets partitioned — the data, the model's layers, or the operations inside a single layer — and each choice trades communication volume against memory savings differently. This page covers the communication mechanics of each strategy: what crosses the interconnect, when, and how much. Distributed Training covers the training-side recipe built on top of these mechanics — ZeRO/FSDP sharding, optimizer state placement, gradient accumulation — and is the page to read for how a framework actually configures and combines them.
GPUDirect & RDMA
Every transfer that has to bounce through a staging buffer in host memory pays for it twice — once copying into the buffer, once out — and that cost shows up whenever a GPU needs to talk to something other than another GPU on the same NVLink fabric: a network card, an NVMe drive, storage over the network. GPUDirect is NVIDIA's umbrella name for the mechanisms that let those other devices address GPU memory directly instead.
Clusters & Schedulers
A GPU on a shared cluster isn't just requested and used the way a local one is — a scheduler decides which physical devices a job gets, renumbers them from the job's point of view, and (on Kubernetes) treats them as an indivisible resource unless something extra is configured. Getting any of this wrong tends to look like a correctness bug — a job silently touching the wrong device, or "no GPUs available" on a node that clearly has some — rather than an obvious scheduling error.