30 docs tagged with "optimization"

"Blueprint is slow" is the kind of claim that's technically true and mostly useless. It's true in the

Two independent caches matter in a LangChain app, and they solve different problems.

The rest of this section explains how to do things correctly; this page is the shorter list of specific ways not to. Each entry names a symptom, the mechanism that causes it, and the fix — a checklist to scan before believing a benchmark or a kernel is done, not a tutorial.

The compilation phase translates preprocessed C++ code into assembly language. This is where syntax checking, semantic analysis, optimization, and code generation happen.

constexpr indicates values or functions can be evaluated at compile-time, enabling compile-time computation and optimization.

Overview

The model scores 95% accuracy in the notebook — and returns nonsense in production. This is one of the most common, most preventable failure modes in applied vision, and it's almost never the model's fault: it's a mismatch between how the training pipeline preprocessed images and how the serving pipeline does.

Nearly every model in this knowledge base is trained the same way: compute the gradient of the loss, take a small step in the opposite direction, repeat. The entire difficulty is in choosing how big a step and how much noise to tolerate along the way.

Embed assembly instructions directly in C++ code for performance-critical operations, hardware access, or platform-specific features unavailable in C++.

inline suggests the compiler replace function calls with function body, eliminating call overhead. Modern compilers decide automatically.

Occupancy Tuning showed that giving a thread several independent accumulators can substitute for occupancy — the scheduler fills a dependent chain's latency with another chain's independent work instead of waiting on another resident warp. This page generalizes that idea beyond hiding latency for its own sake: instruction-level parallelism, loop unrolling, cheaper instruction choices, and avoiding instruction sequences the hardware doesn't handle well are all ways to make the instructions a thread already issues cost less or overlap better, independent of how many warps are resident.

Every kernel launch has a fixed cost — driver overhead to enqueue it, and a round trip through global memory for its inputs and outputs — that has nothing to do with how much useful arithmetic the kernel does. A chain of small, bandwidth-bound elementwise kernels can spend more of its wall-clock time paying that fixed cost, repeatedly, than doing the actual computation, which is what makes fusing them into a single launch — or avoiding the separate launches altogether — a real optimization rather than a stylistic preference.

The loss is the only thing the model actually optimises. Every other design choice — architecture, optimiser, regularisation — is in service of minimising this one number. Choose it carelessly and the model will optimise exactly what you asked for, which is often not what you meant.

Global Memory and Coalescing establishes the 32-byte-sector model and works out the arithmetic for coalesced, strided, and AoS access patterns. This page turns that model into an ordered checklist for a memory-bound kernel: which lever to pull first, which to skip, and why the order matters more than any individual technique.

noexcept specifies that a function won't throw exceptions, enabling optimizations and stronger guarantees.

The Register File and Occupancy derives occupancy as a fixed calculation against three hardware limits — registers, shared memory, and block/thread slots — and works a full example through all three. This page is the tuning counterpart: given that calculation, which levers actually move it, what raising occupancy buys in practice, and — just as important — where it stops buying anything at all.

Raising the optimization level is the one build change that routinely alters what a firmware does. Not what it does more quickly — what it does. A delay loop disappears. A register write that was there at -O0 is gone at -O2. Code that worked for two years starts failing, and nothing in the source changed.

Why this matters

Plain gradient descent takes the same fixed-size step in every direction, every time, regardless of the loss surface's shape. Everything covered on this page is a way of using information from past gradients to take smarter steps — and by the mid-2010s, one method (Adam) had absorbed most of these ideas and become the default nearly everyone reaches for first.

Tools for finding performance bottlenecks: CPU time, cache misses, branch mispredictions. Measure first, then optimize.

Every algorithm so far has taken a plain gradient step on the policy — and a plain gradient step, if it's too large, can destroy the policy in a way a large supervised-learning update never does. Proximal Policy Optimization is the algorithm most production RL systems actually run, and its entire design is built around preventing exactly that failure.

Tensor Cores laid out what tensor cores are and what makes a kernel eligible to reach them; this page is about actually writing code that does — the wmma warp-level API, the two facts about it that trip almost everyone up the first time, and why the code you write here is closer to a teaching exercise than to what a shipped GEMM looks like.

Almost every optimization in this section works through the C++ source and trusts nvcc to generate good machine code from it. Occasionally that trust runs out — an instruction has no intrinsic, or the compiler's chosen code sequence needs to be inspected or overridden directly — and the only way forward is reading or writing below the C++ level. This page covers both: reading the SASS a kernel actually compiles to, and, rarely, writing PTX by hand to reach an instruction the compiler won't emit on its own.

Understanding compiler-generated assembly helps verify optimizations, debug performance issues, and understand low-level behavior.

Warp Execution and Divergence established the cost model: a warp pays for every path its 32 lanes collectively take, sequentially, so a branch only costs extra when lanes within the same warp disagree. This page is the applied counterpart — given that rule, what actually removes the cost in real kernels.

Memory Access Optimization stops once accesses are coalesced and vectorized, because coalescing only fixes how efficiently a kernel fetches the bytes it asks for — it does nothing about a kernel that asks DRAM for the same bytes over and over. Shared-memory tiling attacks that second problem directly: load each piece of data into on-chip shared memory once, then let every thread that needs it read it from there instead of going back to DRAM. This page derives why that reuse matters arithmetically and builds the tiled kernel that makes it concrete.

Shared Memory Tiling ends on the bubble its own sgemmTiled kernel still has: every iteration of the tile loop is load → syncthreads() → compute → syncthreads(), and that structure forces the memory system and the ALUs to take turns rather than work at the same time. Software pipelining removes the turn-taking by starting the next tile's load before the current tile's compute has finished, so the two phases run concurrently across iterations instead of serially within one.

Pointers of different types cannot point to the same memory (with exceptions). Enables compiler optimizations but causes undefined behavior when violated.

Tuning a kernel without a loop around the work turns into guessing: try something that sounds plausible, rerun, eyeball whether it got faster, repeat. The workflow that actually converges is narrower than that — profile to find the one resource the kernel is actually waiting on, apply only the fix that targets that resource, re-measure to confirm the fix worked and see what limiter is binding now, and stop once further gains are no longer worth the effort. Every other page in this folder is a toolbox entry for one step of this loop, not a replacement for it.

volatile tells compiler that a variable can change unexpectedly (hardware, interrupts, other threads). Prevents certain optimizations. Not for thread synchronization - use atomics instead.