GPU & Accelerators
A GPU is a throughput machine bolted onto a latency machine: thousands of simple cores, oversubscribed with far more threads than can run at once, trading single-thread speed for the ability to hide memory latency behind other work. Almost every performance question in this section reduces to the same one — did you keep the memory system busy, or is the chip sitting idle waiting on a load.
This section goes deep on CUDA — the programming model, the memory hierarchy, execution and synchronization, the runtime, and kernel optimization — then widens out to what production work actually uses: libraries, profiling tools, multi-GPU scaling, vendor-neutral portability layers, and NPU/inference accelerators. It does not cover graphics or rendering; for how a game engine's render pipeline is architected and scheduled onto the GPU, see Render thread model in the Unreal Engine section.
Folders 00–02 build the mental model (why GPUs look the way they do, and the hardware underneath). Folders 03–07 are CUDA proper — the programming model, memory, execution, runtime, and optimization. Folders 08–10 are what you reach for in production — libraries, profiling tools, and multi-GPU scaling. Folders 11–13 close out portability, inference accelerators, and worked kernels. Every folder from 03 onward is self-contained — you can start at any of them once you have the 00–02 grounding.
Three learning paths
| Path | Sequence | What you can do at the end |
|---|---|---|
| Write fast CUDA kernels | CUDA Programming Model → Memory Model → Execution → Kernel Optimization → Applied Kernels | Write, launch, and systematically optimize a CUDA kernel |
| Understand the hardware | Foundations → Hardware Architecture → Tooling & Profiling | Explain why a GPU is fast, and read its profiler output |
| Deploy models on accelerators | Overview → Libraries & Ecosystem → NPUs & Inference Accelerators | Pick the right accelerator and library for a deployment target |
Sections
| Section | What it covers |
|---|---|
| Overview | Why GPUs exist, how they compare to CPUs and NPUs, the accelerator landscape, and when not to reach for a GPU |
| Parallel Computing Foundations | Flynn's taxonomy, Amdahl/Gustafson, latency hiding, arithmetic intensity, and the parallel patterns vocabulary |
| GPU Hardware Architecture | GPCs, SMs, warp schedulers, register files, caches, tensor cores, and the NVIDIA architecture generations |
| CUDA Programming Model | Toolkit setup, your first kernel, threads/blocks/grids, launch configuration, and the compilation model |
| CUDA Memory Model | Global, shared, constant, and register memory; coalescing; bank conflicts; unified memory; async data movement |
| Execution and Synchronization | Warp divergence, independent thread scheduling, warp-level primitives, cooperative groups, and atomics |
| CUDA Runtime and APIs | Runtime vs driver API, device management, streams and events, CUDA graphs, and error handling |
| Kernel Optimization | The measure-classify-fix workflow, occupancy tuning, memory access and instruction-level optimization, tensor cores |
| Libraries and Ecosystem | cuBLAS, cuDNN, Thrust, CUB, CUTLASS, NCCL, and the Python stack (CuPy, Numba, Triton, PyTorch extensions) |
| Tooling, Profiling, and Debugging | CMake builds, Nsight Systems and Compute, cuda-gdb and Compute Sanitizer, and roofline analysis in practice |
| Multi-GPU and Scaling | Multi-GPU basics, peer-to-peer and NVLink, NCCL collectives, parallelism strategies, and cluster scheduling |
| Portable and Vendor-Neutral | HIP/ROCm, SYCL/oneAPI, OpenCL, OpenMP/OpenACC offload, Vulkan/DirectX compute, Metal, and WebGPU |
| NPUs and Inference Accelerators | Systolic arrays, Google TPU, edge NPUs, Jetson/DLA, quantization, TensorRT, ONNX Runtime, and OpenVINO |
| Applied Kernels and Patterns | Progressive optimizations of reduction, scan, matrix multiply, histogram, sorting, softmax, and FlashAttention |
Conventions used here
- CUDA C++ is the default language throughout the section and appears in
cppcode fences. - Python is confined to Libraries and Ecosystem (CuPy, Numba, Triton, PyTorch extensions), plus the toolkit-installation page.
- Every compute-capability requirement is called out in a
:::note, since features like thread block clusters and independent thread scheduling only exist from a given architecture generation onward. - Every performance number states the GPU it was measured on — a speedup on one architecture generation does not transfer to another.