Optimization Workflow
Tuning a kernel without a loop around the work turns into guessing: try something that sounds plausible, rerun, eyeball whether it got faster, repeat. The workflow that actually converges is narrower than that — profile to find the one resource the kernel is actually waiting on, apply only the fix that targets that resource, re-measure to confirm the fix worked and see what limiter is binding now, and stop once further gains are no longer worth the effort. Every other page in this folder is a toolbox entry for one step of this loop, not a replacement for it.
Occupancy Tuning
The Register File and Occupancy derives occupancy as a fixed calculation against three hardware limits — registers, shared memory, and block/thread slots — and works a full example through all three. This page is the tuning counterpart: given that calculation, which levers actually move it, what raising occupancy buys in practice, and — just as important — where it stops buying anything at all.
Memory Access
Global Memory and Coalescing establishes the 32-byte-sector model and works out the arithmetic for coalesced, strided, and AoS access patterns. This page turns that model into an ordered checklist for a memory-bound kernel: which lever to pull first, which to skip, and why the order matters more than any individual technique.
Shared Memory Tiling
Memory Access Optimization stops once accesses are coalesced and vectorized, because coalescing only fixes how efficiently a kernel fetches the bytes it asks for — it does nothing about a kernel that asks DRAM for the same bytes over and over. Shared-memory tiling attacks that second problem directly: load each piece of data into on-chip shared memory once, then let every thread that needs it read it from there instead of going back to DRAM. This page derives why that reuse matters arithmetically and builds the tiled kernel that makes it concrete.
Instruction-Level
Occupancy Tuning showed that giving a thread several independent accumulators can substitute for occupancy — the scheduler fills a dependent chain's latency with another chain's independent work instead of waiting on another resident warp. This page generalizes that idea beyond hiding latency for its own sake: instruction-level parallelism, loop unrolling, cheaper instruction choices, and avoiding instruction sequences the hardware doesn't handle well are all ways to make the instructions a thread already issues cost less or overlap better, independent of how many warps are resident.
Reducing Divergence
Warp Execution and Divergence established the cost model: a warp pays for every path its 32 lanes collectively take, sequentially, so a branch only costs extra when lanes within the same warp disagree. This page is the applied counterpart — given that rule, what actually removes the cost in real kernels.
Fusion & Launch Overhead
Every kernel launch has a fixed cost — driver overhead to enqueue it, and a round trip through global memory for its inputs and outputs — that has nothing to do with how much useful arithmetic the kernel does. A chain of small, bandwidth-bound elementwise kernels can spend more of its wall-clock time paying that fixed cost, repeatedly, than doing the actual computation, which is what makes fusing them into a single launch — or avoiding the separate launches altogether — a real optimization rather than a stylistic preference.
Programming Tensor Cores
Tensor Cores laid out what tensor cores are and what makes a kernel eligible to reach them; this page is about actually writing code that does — the wmma warp-level API, the two facts about it that trip almost everyone up the first time, and why the code you write here is closer to a teaching exercise than to what a shipped GEMM looks like.
Software Pipelining
Shared Memory Tiling ends on the bubble its own sgemmTiled kernel still has: every iteration of the tile loop is load → syncthreads() → compute → syncthreads(), and that structure forces the memory system and the ALUs to take turns rather than work at the same time. Software pipelining removes the turn-taking by starting the next tile's load before the current tile's compute has finished, so the two phases run concurrently across iterations instead of serially within one.
PTX & Inline Asm
Almost every optimization in this section works through the C++ source and trusts nvcc to generate good machine code from it. Occasionally that trust runs out — an instruction has no intrinsic, or the compiler's chosen code sequence needs to be inspected or overridden directly — and the only way forward is reading or writing below the C++ level. This page covers both: reading the SASS a kernel actually compiles to, and, rarely, writing PTX by hand to reach an instruction the compiler won't emit on its own.
Antipatterns
The rest of this section explains how to do things correctly; this page is the shorter list of specific ways not to. Each entry names a symptom, the mechanism that causes it, and the fix — a checklist to scan before believing a benchmark or a kernel is done, not a tutorial.