14 docs tagged with "npu"

Every deployment path covered so far in this folder eventually hands a graph to something that turns it into device code, and up to now that "something" has mostly been a fixed toolkit: TensorRT's builder, OpenVINO's plugins, a vendor NPU SDK. This page steps back and looks at the compilers underneath those toolkits and behind the frameworks themselves — what a graph compiler actually does that a kernel library doesn't, and the handful of stacks (XLA, TVM, MLIR, and torch.compile) that show up across nearly every deployment target in this section.

A modern laptop, phone, or server node contains all three of these, and they are not three points on a speed scale. They are three different answers to the question "how much of this chip should be general-purpose?" The CPU keeps every option open and pays for it in area and energy. The GPU gives up per-thread cleverness to buy arithmetic width, but stays fully programmable — you can still write an arbitrary kernel. The NPU gives up general programmability as well, hard-wiring a small set of tensor operations at fixed precisions, and gets back an energy-per-operation figure neither of the others can approach.

Every other page in this folder covers one piece of the deployment problem in depth — a device family, a runtime, a compiler, a quantization technique. This page is the one that ties them together into an order of operations: which decisions to make first, what to check before committing to a target, and what to verify before calling a deployment done. It is deliberately a procedure rather than a survey — the pages it links to already carry the depth, and repeating that depth here would only get it out of sync with the pages that own it.

"NPU" on a phone, laptop, or camera SoC does not name one architecture — it names a family of fixed-function inference engines from different vendors, each with its own SDK, its own supported precisions, and its own idea of how much of the operator set it covers. What Is an NPU covered why this class of hardware exists at all; this page is the vendor-by-vendor reference for the ones you're actually likely to target.

The Tensor Processing Unit is what happens when the weight-stationary systolic array from Systolic Arrays and Dataflow is scaled up to a datacenter training and inference accelerator, with a compiler stack and an interconnect built around it from the start. It is worth its own page separately from the general dataflow discussion because using a TPU well means accepting a programming model that looks nothing like CUDA: you do not write kernels for it at all.

A GPU is a throughput machine bolted onto a latency machine: thousands of simple cores, oversubscribed with far more threads than can run at once, trading single-thread speed for the ability to hide memory latency behind other work. Almost every performance question in this section reduces to the same one — did you keep the memory system busy, or is the chip sitting idle waiting on a load.

Jetson is not a scaled-down GPU with different rules — it is a full CUDA-capable GPU on the same die as an Arm CPU, so everything in folders 03 through 07 of this section applies to it directly memory is physically shared between CPU and GPU, the power envelope is fixed and small, and a second, fixed-function inference engine — the DLA — sits alongside the GPU on the same package.

Getting a model off a training framework and onto an arbitrary piece of inference hardware needs a common description both sides agree on. ONNX is that description: a standardized, framework-neutral way to write down a model graph so that PyTorch, and separately TensorRT, and separately a phone's NPU compiler, can all read the same file and agree on what it means. What actually executes that file is a different question, and conflating the two is the single most common confusion this page exists to clear up.

Most of the deployment stacks in this folder are strongest on hardware from one vendor, and OpenVINO's vendor is Intel. If the deployment target is an Intel CPU, an Intel integrated GPU, or the NPU built into a recent Intel Core Ultra laptop chip, OpenVINO is usually the toolkit that gets the most performance out of that hardware with the least fighting — it is Intel's own inference stack, tuned against Intel's own silicon, and it is the natural first thing to reach for once the deployment machine is known to be an Intel client device.

An NPU's MAC array (see What Is an NPU?) is built around integer arithmetic first and floating point second, if at all. Getting a model onto that hardware efficiently means converting its weights and activations from floating point into integers in a way that a fixed set of scale and offset numbers can undo well enough that the model still works. That conversion — quantization — is a distinct engineering discipline from anything in ordinary model training, with its own vocabulary, its own failure modes, and its own tooling, and this page is that vocabulary.

What Is an NPU established that an NPU's efficiency comes from removing per-instruction overhead, not from having more MAC units than a GPU. This page is about the mechanism that actually achieves that: the systolic array, and the small set of dataflow patterns — weight-stationary, output-stationary, row-stationary — that decide what stays resident in each cell versus what streams past it. Which pattern a piece of hardware picks is not a minor implementation detail; it is the single biggest factor in how much energy that hardware spends moving data around.

The central fact to hold onto about TensorRT is that it is a compiler, not a runtime library you call into layer by layer. Given a model graph and the exact shapes, precisions, and target GPU you tell it about, it benchmarks candidate kernel implementations for every layer, picks the fastest ones for that specific hardware, fuses what it can, and emits a serialized engine — a compiled artifact, not a portable model file. That one fact explains everything else on this page: why building an engine is slow (it is a search over kernel candidates, not a translation), why the resulting engine is fast (every layer runs the kernel TensorRT found to be fastest on that GPU, for those shapes), and why the engine does not travel to a different GPU, a different TensorRT version, or often even a different driver.

Once you accept that some of your work belongs on a throughput engine, you have to pick one, and the market offers far more options than "NVIDIA or not". There are discrete GPUs on a PCIe slot, GPUs integrated into the same die as the CPU, phone-class GPUs paired with NPUs, datacenter training and inference ASICs reachable only through a compiler, and FPGAs where you describe the datapath yourself. They differ enormously in peak throughput — and that difference is almost never what decides the outcome.

Every accelerator in this section so far has been a variation on "more programmable cores, running in parallel." An NPU (neural processing unit) is a different move entirely: instead of adding parallel general-purpose lanes, it removes almost all of the general-purpose machinery and replaces it with a dataflow of multiply-accumulate (MAC) cells wired specifically for tensor arithmetic. CPU vs GPU vs NPU already introduced this as the third design point; this page works through what that design point actually buys and what it costs.