2 docs tagged with "throughput"

Latency and throughput sound like the same idea measured two ways, but a GPU treats them as almost unrelated design targets. Latency is how long one memory request takes to come back; throughput is how many bytes per second the memory system can sustain in steady state. A single DRAM access on a modern GPU takes several hundred nanoseconds — not meaningfully faster than it was a decade ago — yet the same hardware sustains terabytes per second in aggregate. The only way to reconcile a slow individual request with a fast aggregate rate is to have an enormous number of requests outstanding at once, and that single fact is why the CUDA programming model insists you expose thousands of threads instead of a handful.

The useful question is not "why is a GPU faster than a CPU" — it usually isn't. A single CPU core will finish one dependent chain of instructions sooner than any GPU will, and it will do it on branchy, pointer-chasing, irregular code that a GPU handles badly. The real question is how a fixed transistor budget gets spent. A CPU spends most of its area on machinery that makes one instruction stream go fast: out-of-order scheduling, register renaming, branch prediction, and a deep cache hierarchy that hides DRAM latency from a handful of threads. A GPU deletes almost all of that and spends the reclaimed area on arithmetic units, then keeps them busy by oversubscribing the machine with far more threads than can execute in any one cycle.