6 docs tagged with "inference"

Every other page in this folder covers one piece of the deployment problem in depth — a device family, a runtime, a compiler, a quantization technique. This page is the one that ties them together into an order of operations: which decisions to make first, what to check before committing to a target, and what to verify before calling a deployment done. It is deliberately a procedure rather than a survey — the pages it links to already carry the depth, and repeating that depth here would only get it out of sync with the pages that own it.

The model scores 95% accuracy in the notebook — and returns nonsense in production. This is one of the most common, most preventable failure modes in applied vision, and it's almost never the model's fault: it's a mismatch between how the training pipeline preprocessed images and how the serving pipeline does.

Making the model cheap enough to serve, without quietly making it wrong. Every optimisation technique on this page trades accuracy, latency, or memory for one another — measure all three, before and after, or the "optimisation" is a guess.

Getting a model off a training framework and onto an arbitrary piece of inference hardware needs a common description both sides agree on. ONNX is that description: a standardized, framework-neutral way to write down a model graph so that PyTorch, and separately TensorRT, and separately a phone's NPU compiler, can all read the same file and agree on what it means. What actually executes that file is a different question, and conflating the two is the single most common confusion this page exists to clear up.

Choosing between an overnight batch job and a 50-millisecond synchronous API is not a minor implementation detail — it changes almost every other decision downstream, from infrastructure to model architecture to error handling. The serving pattern should be chosen deliberately, before the model is designed, not bolted on after.

The central fact to hold onto about TensorRT is that it is a compiler, not a runtime library you call into layer by layer. Given a model graph and the exact shapes, precisions, and target GPU you tell it about, it benchmarks candidate kernel implementations for every layer, picks the fastest ones for that specific hardware, fuses what it can, and emits a serialized engine — a compiled artifact, not a portable model file. That one fact explains everything else on this page: why building an engine is slow (it is a search over kernel candidates, not a translation), why the resulting engine is fast (every layer runs the kernel TensorRT found to be fastest on that GPU, for those shapes), and why the engine does not travel to a different GPU, a different TensorRT version, or often even a different driver.