The non-linearity between layers is what makes depth meaningful at all — From Perceptron to MLP proved that a stack of purely linear layers collapses into a single linear layer. But not all non-linearities are equal: the shape of an activation's derivative determines whether gradients survive a deep network or vanish before reaching the early layers.
Every parameter in a deep network needs its own gradient, and a network can have millions of them. Backpropagation is the algorithm that computes every single one, applying the chain rule from Calculus and Gradients systematically backwards through the computational graph — at roughly the same cost as one forward pass. That efficiency is the entire reason deep learning became computationally feasible.
A GPU can process a batch of data in milliseconds — and then sit idle for seconds waiting for the next batch to be loaded and preprocessed from disk. The input pipeline is where a surprising fraction of real training time actually goes, and getting it wrong silently turns an expensive GPU into an expensive way to wait for a CPU.
The loss isn't going down, and unlike a typical software bug, there's no stack trace pointing at the problem — the model runs, produces numbers, and those numbers are simply wrong in a way Python's error messages have nothing to say about. The fix is to debug in a fixed order, because each step rules out an entire category of causes before moving to the next.
One GPU stops being enough for three distinct reasons — training is too slow, the model doesn't fit in memory, or the batch size that would actually converge well doesn't fit either — and each has a genuinely different fix. Reaching for the wrong one wastes both engineering effort and compute budget.
A neural network is a function composition, layer feeding layer, and the graph of that composition is exactly what gets differentiated to train the network. Writing the forward pass explicitly as a graph of small, primitive operations is what turns "compute gradients" from a calculus exercise into a completely mechanical procedure — this page establishes that graph view before Backpropagation differentiates it.
In 1958, Frank Rosenblatt built a model that could learn simple logical patterns from data — and in 1969, Minsky and Papert proved it could never learn XOR, a result that froze neural network research for over a decade. The fix, once found, was a single conceptual change: stack more than one layer, with something non-linear between them.
The fastest way to make training slower is to leave the GPU waiting on the CPU — and the second fastest way to make it faster, after fixing that, is to stop computing every number with more precision than the task actually needs. Most real speedups come from these two unglamorous facts, not from a cleverer algorithm.
The learning rate is the single hyperparameter that most determines whether training works at all, and holding it fixed for the entire run is rarely the best choice. Every schedule on this page is a variation of the same idea: large steps early, when you're far from a good solution and want to explore quickly, small steps late, when you're close and want to settle precisely.
How big should the model be? The classical answer from Bias-Variance Tradeoff — bigger risks overfitting past some point — turns out to be incomplete for the over-parameterised networks that now dominate deep learning, where a second, deeper descent in test error can appear past the point where the classical U-curve says things should be getting worse.
Networks deeper than twenty or so layers were, for years, essentially untrainable — every layer's input distribution kept shifting as the layers below it updated, forcing each layer to constantly re-adapt to a moving target. Batch normalisation, and the family of normalisation layers it started, fixed this directly by re-standardising activations at every layer, which is what let networks scale to hundreds of layers.
Plain gradient descent takes the same fixed-size step in every direction, every time, regardless of the loss surface's shape. Everything covered on this page is a way of using information from past gradients to take smarter steps — and by the mid-2010s, one method (Adam) had absorbed most of these ideas and become the default nearly everyone reaches for first.
Every gradient in the previous eleven pages was derived and coded by hand. From here on, a framework does that work — but only because it implements exactly the mechanism Backpropagation already described: recording a computational graph as operations run, then walking it backward. Autograd is the manual backward pass, automated and generalised to arbitrary graphs.
A modern network routinely has more parameters than training examples — by classical statistical intuition, this should guarantee catastrophic overfitting. It usually doesn't, and the regularisation techniques on this page are less about shrinking weights (as in Overfitting and Regularization's classical L1/L2 story) and more about injecting noise or stopping early.
In 2015, researchers found something strange: a 56-layer network had higher training error than a 20-layer network on the same task — not overfitting (that would show as a validation gap), but a genuine failure to optimise the deeper network at all. The fix, adding the input back to a layer's output, was a two-line change that took feasible network depth from roughly twenty layers to over a thousand.
Every deep learning project, regardless of architecture or task, runs the same twenty-odd lines at its core: forward, compute loss, zero the gradients, backward, step. That order is not negotiable — get it wrong and training either does nothing or does something subtly incorrect, and both failure modes tend to fail quietly rather than crash loudly.
For two decades, networks deeper than a handful of layers simply refused to train — not because the architecture was wrong, but because the gradient signal reaching the earliest layers had either shrunk to numerical zero or grown to numerical infinity by the time it arrived there. Understanding exactly why this happens is what makes the fixes (initialisation, activations, normalisation, residual connections) make sense as one coherent story rather than a grab-bag of tricks.
Before the first gradient is ever computed, a choice has already been made that decides whether training has any chance of working: how the weights start. Get it wrong and the signal either dies (shrinks to zero within a few layers) or explodes (grows without bound) before a single useful gradient update happens.