9 docs tagged with "runtime"

A single kernel launch costs the CPU roughly 3–10 µs of driver-side work, independent of how much the kernel actually does. A pipeline that issues 50 small kernels per iteration can spend more time launching work than the GPU spends computing it, and Streams and Concurrency doesn't fix that — streams reorder and overlap launches, they don't reduce their count. A CUDA graph captures a whole sequence of operations once and replays it as a single launch, collapsing 50 dispatches into one.

A host process can see more than one GPU, and CUDA never guesses which one a call should target — it operates against whichever device is current on the calling thread, a piece of state the program has to set itself. Getting this wrong doesn't usually crash; it silently allocates memory or launches kernels on the wrong GPU, or leaves a multi-threaded program with each thread quietly disagreeing about which device it's using. See Error Handling and Checking for what CUDA_CHECK does with the status these calls return.

Most kernels launch from the host with a grid size chosen before any device-side work has happened, which is a poor fit for problems whose parallelism is only known once the GPU has started computing — a mesh that needs refining only in some regions, a tree whose branching factor varies by node, a search whose frontier grows unpredictably. Dynamic parallelism lets a kernel launch further kernels directly from the device, so the grid for the next phase can be sized from data the first phase just produced, without a round-trip through the host.

A CUDA API call that fails almost never fails where the mistake happened. Because most of the runtime is asynchronous, a kernel launch or a copy can return immediately with cudaSuccess on the host side while the actual work — and the actual error, if there is one — hasn't executed on the GPU yet. Unchecked, that error surfaces as a failure on some unrelated call several lines or several function calls later, which is why every other page in this section wraps runtime calls in a checking macro rather than trusting a bare return value.

A cudaEventt is a marker that can be dropped into a stream and later queried, waited on, or used to measure elapsed time between two points — it's the mechanism behind both accurate kernel timing and dependencies between streams that don't require the host to get involved. Both uses matter here: naive host-clock timing of GPU work produces numbers that look plausible and are wrong, and coordinating streams without a host round-trip is what makes the concurrency from Streams and Concurrency composable into a real pipeline. See Error Handling and Checking for what CUDACHECK does with the status these calls return.

cudaMalloc is the allocator every earlier example reached for, and for a program that allocates once at startup and frees once at exit, it's the right tool. It stops being the right tool the moment allocation moves inside a loop, because cudaMalloc and cudaFree are synchronizing, device-wide operations — they can take tens of microseconds each, which is invisible in a single call and devastating when it happens every iteration of a hot loop. The APIs on this page exist to give allocation a shape that matches how a program actually uses memory: padded for coalescing, ordered in a stream, pooled, or — rarely — managed as raw virtual address space.

A single GPU is often shared by more processes than it has obvious ways to divide itself among. The default sharing mechanism, time-slicing, works but wastes capacity on small kernels; two other mechanisms exist to do better, and they solve different problems. Multi-Process Service (MPS) lets independent processes' kernels run concurrently instead of merely taking turns; Multi-Instance GPU (MIG) physically partitions the hardware so processes don't share anything at all. Choosing between them means understanding what each one isolates and what it doesn't.

Every CUDA C++ example so far — >> launches, cudaMalloc, cudaMemcpy — has gone through the runtime API, the high-level interface linked in as cudart and initialized implicitly the first time a program touches the GPU. Underneath it sits the driver API (cuda.h, linked as cuda), a lower-level, explicit interface that the runtime itself is built on. Almost nothing in application code needs the driver API directly, but understanding what the runtime is hiding explains a class of errors ("invalid device context") that only make sense once you know a context exists at all.

Pinned Memory and Host Transfers split a transfer into four chunks, each on its own stream, to overlap H2D copies with kernel execution — but it left unanswered exactly what a stream guarantees, why that overlap can silently fail to happen even with the code written correctly, and how to reason about ordering across streams on purpose. That's this page.