2 docs tagged with "nccl"

The NCCL page covers the API — communicators, the collective calls, stream integration, grouped calls. This page covers what happens underneath an ncclAllReduce call: the ring and tree algorithms NCCL chooses between, the cost model that explains why ring all-reduce scales the way it does, and how a training loop overlaps communication with compute instead of paying for it serially. It doesn't repeat the API surface — link there for ncclCommInitRank, the collectives table, or ncclGroupStart/ncclGroupEnd.

NCCL (NVIDIA Collective Communications Library) is the library that moves data between GPUs — within a node over NVLink or PCIe, and across nodes over the network — through a small set of collective operations borrowed from the MPI world setting up a communicator, the collectives themselves, and how a call integrates with a stream. Collectives with NCCL covers the harder half — ring versus tree algorithm selection, and overlapping communication with gradient computation in a real training loop — and builds directly on the vocabulary defined here.