What Is an NPU?
Every accelerator in this section so far has been a variation on "more programmable cores, running in parallel." An NPU (neural processing unit) is a different move entirely: instead of adding parallel general-purpose lanes, it removes almost all of the general-purpose machinery and replaces it with a dataflow of multiply-accumulate (MAC) cells wired specifically for tensor arithmetic. CPU vs GPU vs NPU already introduced this as the third design point; this page works through what that design point actually buys and what it costs.
Systolic Arrays
What Is an NPU established that an NPU's efficiency comes from removing per-instruction overhead, not from having more MAC units than a GPU. This page is about the mechanism that actually achieves that: the systolic array, and the small set of dataflow patterns — weight-stationary, output-stationary, row-stationary — that decide what stays resident in each cell versus what streams past it. Which pattern a piece of hardware picks is not a minor implementation detail; it is the single biggest factor in how much energy that hardware spends moving data around.
Google TPU
The Tensor Processing Unit is what happens when the weight-stationary systolic array from Systolic Arrays and Dataflow is scaled up to a datacenter training and inference accelerator, with a compiler stack and an interconnect built around it from the start. It is worth its own page separately from the general dataflow discussion because using a TPU well means accepting a programming model that looks nothing like CUDA: you do not write kernels for it at all.
Edge NPUs
"NPU" on a phone, laptop, or camera SoC does not name one architecture — it names a family of fixed-function inference engines from different vendors, each with its own SDK, its own supported precisions, and its own idea of how much of the operator set it covers. What Is an NPU covered why this class of hardware exists at all; this page is the vendor-by-vendor reference for the ones you're actually likely to target.
Jetson & DLA
Jetson is not a scaled-down GPU with different rules — it is a full CUDA-capable GPU on the same die as an Arm CPU, so everything in folders 03 through 07 of this section applies to it directly memory is physically shared between CPU and GPU, the power envelope is fixed and small, and a second, fixed-function inference engine — the DLA — sits alongside the GPU on the same package.
Quantization
An NPU's MAC array (see What Is an NPU?) is built around integer arithmetic first and floating point second, if at all. Getting a model onto that hardware efficiently means converting its weights and activations from floating point into integers in a way that a fixed set of scale and offset numbers can undo well enough that the model still works. That conversion — quantization — is a distinct engineering discipline from anything in ordinary model training, with its own vocabulary, its own failure modes, and its own tooling, and this page is that vocabulary.
TensorRT
The central fact to hold onto about TensorRT is that it is a compiler, not a runtime library you call into layer by layer. Given a model graph and the exact shapes, precisions, and target GPU you tell it about, it benchmarks candidate kernel implementations for every layer, picks the fastest ones for that specific hardware, fuses what it can, and emits a serialized engine — a compiled artifact, not a portable model file. That one fact explains everything else on this page: why building an engine is slow (it is a search over kernel candidates, not a translation), why the resulting engine is fast (every layer runs the kernel TensorRT found to be fastest on that GPU, for those shapes), and why the engine does not travel to a different GPU, a different TensorRT version, or often even a different driver.
ONNX & Runtimes
Getting a model off a training framework and onto an arbitrary piece of inference hardware needs a common description both sides agree on. ONNX is that description: a standardized, framework-neutral way to write down a model graph so that PyTorch, and separately TensorRT, and separately a phone's NPU compiler, can all read the same file and agree on what it means. What actually executes that file is a different question, and conflating the two is the single most common confusion this page exists to clear up.
OpenVINO
Most of the deployment stacks in this folder are strongest on hardware from one vendor, and OpenVINO's vendor is Intel. If the deployment target is an Intel CPU, an Intel integrated GPU, or the NPU built into a recent Intel Core Ultra laptop chip, OpenVINO is usually the toolkit that gets the most performance out of that hardware with the least fighting — it is Intel's own inference stack, tuned against Intel's own silicon, and it is the natural first thing to reach for once the deployment machine is known to be an Intel client device.
Compiler Stacks
Every deployment path covered so far in this folder eventually hands a graph to something that turns it into device code, and up to now that "something" has mostly been a fixed toolkit: TensorRT's builder, OpenVINO's plugins, a vendor NPU SDK. This page steps back and looks at the compilers underneath those toolkits and behind the frameworks themselves — what a graph compiler actually does that a kernel library doesn't, and the handful of stacks (XLA, TVM, MLIR, and torch.compile) that show up across nearly every deployment target in this section.
Deploying
Every other page in this folder covers one piece of the deployment problem in depth — a device family, a runtime, a compiler, a quantization technique. This page is the one that ties them together into an order of operations: which decisions to make first, what to check before committing to a target, and what to verify before calling a deployment done. It is deliberately a procedure rather than a survey — the pages it links to already carry the depth, and repeating that depth here would only get it out of sync with the pages that own it.