10 docs tagged with "parallelism"

Buying a bigger GPU, or more of them, does not buy a proportionally bigger speedup, and the reason has nothing to do with the hardware being slow. It has to do with the fraction of the program that was never made parallel in the first place. Two laws describe the two ways to think about that fraction — one holds the problem size fixed and asks how fast you can finish it, the other holds the time budget fixed and asks how much bigger a problem you can solve — and knowing which one describes your situation changes what "more parallelism" is even supposed to buy you.

Every kernel is limited by one of two things before it is limited by anything else: how fast the device can do arithmetic, or how fast the device can move bytes from DRAM. Which one applies is a property of the kernel's own math, computable on paper before you write a line of CUDA, and it determines almost everything about how you should spend optimization effort afterward. The roofline model is the tool that turns "which one applies" into a single number you can compute and a single plot you can place it on.

Splitting a training job across GPUs means choosing what gets partitioned — the data, the model's layers, or the operations inside a single layer — and each choice trades communication volume against memory savings differently. This page covers the communication mechanics of each strategy: what crosses the interconnect, when, and how much. Distributed Training covers the training-side recipe built on top of these mechanics — ZeRO/FSDP sharding, optimizer state placement, gradient accumulation — and is the page to read for how a framework actually configures and combines them.

Latency and throughput sound like the same idea measured two ways, but a GPU treats them as almost unrelated design targets. Latency is how long one memory request takes to come back; throughput is how many bytes per second the memory system can sustain in steady state. A single DRAM access on a modern GPU takes several hundred nanoseconds — not meaningfully faster than it was a decade ago — yet the same hardware sustains terabytes per second in aggregate. The only way to reconcile a slow individual request with a fast aggregate rate is to have an enormous number of requests outstanding at once, and that single fact is why the CUDA programming model insists you expose thousands of threads instead of a handful.

Knowing a kernel's arithmetic intensity puts it on the roofline plot in theory; knowing whether it is actually memory-bound or compute-bound in practice requires measuring the running kernel, because achieved bandwidth and achieved compute throughput are never the datasheet peaks the paper estimate assumed. This page turns the roofline classification from Arithmetic Intensity and the Roofline Model into a concrete diagnostic you run against a profiler, and adds the case the roofline model doesn't represent at all.

Almost every GPU kernel, however specialized, is built from a small set of recurring data-access shapes. Recognizing which pattern a problem is — before writing any code — tells you how parallelizable it is, what its likely performance limiter will be, and often points directly at a library implementation that already exists and is already tuned. This page names those shapes once, and every later applied-kernel page in this knowledge base assumes you already know these names — "this is a reduction" or "this needs a scan" is meant to carry full meaning by the time you reach folder 13.

Every processor design answers two questions: how many instruction streams does it execute, and how many data streams does each instruction touch. Those two answers are the whole of Flynn's taxonomy, and they matter here because "GPU" is not a single point on that map — a GPU's arithmetic units are driven by an execution model, SIMT, that is easy to mistake for ordinary SIMD vectorization and behaves differently in exactly the cases that matter for correctness and performance.

Before any of the CUDA syntax in the next section makes sense, one structural fact has to be settled: a discrete GPU is a separate computer. It has its own memory, its own processors, and no automatic view of what the CPU is doing — every byte the GPU touches had to arrive there deliberately, and every result has to leave the same way. This page names that structure once, independent of any specific API, so that the CUDA-specific mechanics in Your First Kernel land on a model you already have rather than a pile of new syntax to memorize.

Threads allow programs to perform multiple operations concurrently. C++11 introduced std::thread for portable threading.

A thread pool manages a fixed set of worker threads that execute tasks from a queue, avoiding the overhead of creating and destroying threads repeatedly.