Anatomy of a GPU
A spec sheet lists a GPU as a pile of numbers — core count, clock speed, memory bandwidth — but those numbers only make sense once you know what physical structure they're describing. A GPU is not a bag of independent processors; it's a small number of large, warp-scheduling processors (streaming multiprocessors), each built from smaller replicated pieces, all sharing a common path out to memory. Understanding that structure top to bottom is what turns a spec sheet from marketing copy into something you can reason about.
Streaming Multiprocessor
The SM is the unit everything about GPU performance is ultimately accounted against: occupancy, register pressure, shared-memory capacity, and warp scheduling are all per-SM quantities. Zooming into one SM explains why a block, once scheduled, stays resident on a single SM for its entire lifetime, and why the resources that limit how many blocks can run concurrently are the ones this page enumerates.
Warps & Schedulers
Every SIMT behavior that looks unusual coming from CPU threading — coalescing, divergence, the fact that occupancy is measured in resident warps rather than resident threads — traces back to one hardware fact what it decides among, how fast it can issue, and how to read its behavior back out of a profiler.
Registers & Occupancy
Occupancy is defined in the glossary as the ratio of resident warps to the maximum an SM supports, and Latency, Throughput, and Latency Hiding explains why that ratio matters — resident warps are what supply the concurrent memory requests Little's Law demands. This page is about the other half: occupancy is not a tunable dial, it's the output of a fixed calculation against fixed hardware limits, and the register file is usually the tightest of those limits.
Cache Hierarchy
A GPU's cache hierarchy looks superficially like a CPU's — an L1 per core-analog, a shared L2 behind it — but the access pattern it's optimized for is completely different. A CPU cache is tuned for one thread's temporal and spatial locality; a GPU's L1 and L2 exist to serve tens of thousands of threads issuing memory requests in 32-wide warps, and the granularity at which those requests are actually served is the fact that explains coalescing, wasted bandwidth, and most of what looks like "mysterious" memory performance on a GPU.
Memory & Bandwidth
Every figure in Arithmetic Intensity and the Roofline Model ultimately rests on one number: how many bytes per second a kernel can move between the SMs and DRAM. That number is set by the physical memory technology soldered onto (or stacked next to) the GPU die, and it is very different from the number a datasheet advertises. This page covers what GDDR and HBM actually are, why achieved bandwidth always falls short of the peak figure, and how to measure the one that actually matters for a given kernel.
Tensor Cores
A CUDA core executes one scalar fused-multiply-add per thread per cycle. A tensor core executes an entire small matrix-multiply-accumulate in hardware, cooperatively across a warp, in roughly the same number of cycles — which is why a kernel that reaches them can be an order of magnitude faster than the same arithmetic done on CUDA cores, and why so much of applied deep-learning performance work is really about getting a kernel eligible for tensor cores rather than about tuning ordinary FP32 code.
Architecture Generations
Marketing names like "Ampere" or "Hopper" map to a compute capability number, and that number — not the marketing name — is what actually gates whether a piece of code compiles or runs. This page walks the generations that matter for code written today, listing only what each one added that changes what you can write or how you must write it, and closes with the canonical table this section's other pages point back at whenever they gate a feature behind a specific compute capability.
Compute Capability
Compute capability is the single number that determines what a piece of CUDA code can assume about the GPU it runs on — which instructions exist, which tensor-core precisions are available, how big a thread block cluster can be. Getting the build flags around it wrong is one of the most common ways a CUDA binary that worked on the machine it was built on fails, silently or loudly, on someone else's GPU.
PCIe & NVLink
Everything so far in this section covers bandwidth inside a single GPU — SM to L2, L2 to HBM. The moment a workload needs data on another device, whether that's the host CPU or a second GPU, a completely different and usually much slower link is in the critical path. Which link is available, and at what bandwidth, is not a software choice — it's a property of the physical topology of the machine, and designing a multi-GPU strategy without first knowing that topology is a common source of disappointing scaling.