A GPU can process a batch of data in milliseconds — and then sit idle for seconds waiting for the next batch to be loaded and preprocessed from disk. The input pipeline is where a surprising fraction of real training time actually goes, and getting it wrong silently turns an expensive GPU into an expensive way to wait for a CPU.
PyTorch composes: nearly anything can be built from existing operators. What composition cannot always give you is one kernel. A sequence of PyTorch ops writes every intermediate tensor to global memory and reads it back for the next op, so a chain of cheap elementwise operations spends almost all its time moving data — the problem Kernel Fusion and Launch Overhead covers in general.
Every gradient in the previous eleven pages was derived and coded by hand. From here on, a framework does that work — but only because it implements exactly the mechanism Backpropagation already described: recording a computational graph as operations run, then walking it backward. Autograd is the manual backward pass, automated and generalised to arbitrary graphs.
Every deep learning project, regardless of architecture or task, runs the same twenty-odd lines at its core: forward, compute loss, zero the gradients, backward, step. That order is not negotiable — get it wrong and training either does nothing or does something subtly incorrect, and both failure modes tend to fail quietly rather than crash loudly.