Installing CUDA
Getting nvcc to compile a .cu file and getting a program to actually run on the GPU are two different problems, and most first-time setup failures come from conflating them. Three separate pieces of software have to agree with each other — a kernel-level driver, a toolkit for building code, and a runtime linked into the binary — and version mismatches between them are the single most common reason "it compiled but won't run" happens.
Your First Kernel
Every CUDA program, no matter how large, is built from the same five moves: allocate device memory, copy input in, launch a kernel, copy output back, free what was allocated. SAXPY — y = a*x + y, scalar-times-vector-plus-vector — is small enough to show all five in one file without anything else getting in the way. The rest of this page walks the same file section by section.
Threads, Blocks, Grids
A kernel launch like saxpy>>(...) doesn't just start "some threads" — it starts a precisely structured hierarchy, and the shape of that hierarchy is what lets the same compiled kernel run correctly on a small laptop GPU and a data-center accelerator with an order of magnitude more SMs. Understanding the levels of that hierarchy, and which ones can and can't communicate, is the difference between a kernel that scales and one that only happens to work on the GPU it was tested on.
Thread Indexing
Every thread in a kernel runs the same code, so the only thing that makes it operate on its piece of the data rather than every thread's piece is the index it computes from its own position in the grid. Getting that formula right — and guarding it correctly — is the one piece of CUDA arithmetic that shows up in essentially every kernel, from SAXPY to the applied kernels later in this section.
Launch Configuration
> looks like two arbitrary integers, but each one is a real decision with hardware consequences: block size determines how a block's resources are packed into an SM's fixed budgets, and grid size determines how evenly the total work spreads across the GPU's SMs. Picking both well is mostly a matter of a few rules of thumb plus one API that does the hardware-limit arithmetic for you.
Thread Block Clusters
Blocks are independent by design: no portable synchronization between them, no shared on-chip memory, and no guarantee two blocks even run at the same time. That independence is what lets a kernel scale from a laptop GPU to a data-center one, but it also means algorithms that need a little cross-block cooperation — a bit more shared memory than one block's SM can hold, or a barrier across a handful of blocks — have nowhere to turn. A thread block cluster relaxes exactly that restriction, for a small group of blocks the hardware guarantees will be co-resident on the same GPU Processing Cluster (GPC) at the same time.
Qualifiers
Every function and variable in a CUDA source file needs to answer two questions: which processor does this run on or live on, and who is allowed to call or touch it. C++ alone has no way to express that — global, device, host, shared, constant, and their relatives are CUDA's answer, and getting them right is what makes the rest of the language (templates, references, most of the standard library subset) usable across the host/device boundary at all.
Compilation Model
A single .cu file contains two programs wearing one extension: host C++ that runs on the CPU, and device code that has to end up as instructions a specific GPU can execute. nvcc is the tool that splits those apart, compiles each with the right compiler, and glues the results back into one binary — understanding that split is what makes -arch, -code, and the difference between a build that runs everywhere and one that only runs on the GPU it was built for make sense.
Separate Compilation
nvcc defaults to compiling each .cu file's device code as a self-contained whole, with every device call resolved and inlined within that one translation unit. That default is invisible right up until device code needs to span files, at which point it fails in a way whole-program C++ intuition doesn't predict.