Warp Divergence
Warps and Warp Schedulers established that a warp scheduler issues one instruction to all 32 lanes of a warp at once. That raises an obvious question: what happens when those 32 threads disagree about which instruction to execute next, because a branch condition evaluated differently across lanes? The answer — the warp executes both outcomes and masks off the lanes that don't apply to each one — is the single most important cost model in CUDA kernel design, and getting it precise is the point of this page.
Independent Thread Scheduling
Warp Execution and Divergence described divergence as a cost model: a warp pays for every path its lanes take. Before Volta, divergence was also a scheduling model with sharp edges — the hardware tracked one program counter per warp using an explicit reconvergence stack, and code that assumed lockstep execution within a warp could rely on undocumented but consistent scheduling behavior. Volta replaced that mechanism, and the change is why so many older warp-synchronous idioms are now silently broken rather than merely non-portable.
Warp Primitives
Independent Thread Scheduling explained why lockstep-dependent, volatile-based tricks for exchanging data within a warp no longer work, and why the fix is a family of intrinsics that carry explicit synchronization and an explicit participant mask. This page is that family: the shuffle intrinsics for moving values directly between lanes' registers, the vote intrinsics for asking a yes/no question of the whole warp, and the canonical reduction pattern built from them.
Block Synchronization
The warp-level tools covered so far — divergence handling, independent thread scheduling, the shuffle and vote intrinsics — all operate within a single warp of 32 threads. Most kernels that use shared memory need something coarser: a guarantee that every thread in the whole block, potentially many warps, has reached a point and that everything they wrote before that point is visible to everything they read after it. That guarantee is syncthreads(), and getting its rules exactly right is what stands between a working tiled kernel and one that hangs or reads garbage on some inputs and not others.
Cooperative Groups
Warp-Level Primitives and Block Synchronization both work, but both lean on implicit context there is no implicit lockstep assumption left to break, because every operation states which threads it applies to.
Atomics
Some updates can't wait for a barrier — a histogram bin, a running total, a lock-free counter — because the threads touching the same location aren't at a point where syncthreads() or a group sync() even applies; they need the read-modify-write itself to be indivisible. Atomics provide that: a hardware-guaranteed sequence of read, modify, and write on a single memory location that no other thread's atomic on the same location can interleave with. What atomics don't provide is speed for free — how many threads target the same address, not how many threads issue atomics in total, is what determines whether that guarantee is nearly free or a serialization bottleneck.
Grid-Wide Sync
syncthreads() barriers a block; cluster.sync() barriers a cluster; neither reaches every block in a grid. Some algorithms genuinely need that — a multi-pass iterative solver that must finish writing generation *N* everywhere before any block reads generation *N* for generation *N+1*, for instance — and the usual answer, launching a second kernel between the passes, has real cost when the intermediate state is large and expensive to leave and re-establish. Grid-wide synchronization exists for that case, but it is not simply "a bigger syncthreads()": it comes with a hardware constraint that shapes the whole launch around it.
Reductions & Scans
Summing an array, finding its maximum, counting matches — these collapse many values into one, and doing it efficiently on a GPU means combining values in parallel at every level of the hierarchy rather than serializing down to one thread. Warp-Level Primitives already built the innermost piece, warpReduceSum; this page builds outward from it — warp to block to grid — and then covers the related but distinct problem of a scan, where every intermediate result is wanted, not just the final one.