Every parameter in a deep network needs its own gradient, and a network can have millions of them. Backpropagation is the algorithm that computes every single one, applying the chain rule from Calculus and Gradients systematically backwards through the computational graph — at roughly the same cost as one forward pass. That efficiency is the entire reason deep learning became computationally feasible.
The loss isn't going down, and unlike a typical software bug, there's no stack trace pointing at the problem — the model runs, produces numbers, and those numbers are simply wrong in a way Python's error messages have nothing to say about. The fix is to debug in a fixed order, because each step rules out an entire category of causes before moving to the next.
GANs are famously hard to train, and the failure modes are specific and recognisable. Worse: the standard debugging instinct — watch the loss curve — actively misleads here.
Nearly every model in this knowledge base is trained the same way: compute the gradient of the loss, take a small step in the opposite direction, repeat. The entire difficulty is in choosing how big a step and how much noise to tolerate along the way.
The learning rate is the single hyperparameter that most determines whether training works at all, and holding it fixed for the entire run is rarely the best choice. Every schedule on this page is a variation of the same idea: large steps early, when you're far from a good solution and want to explore quickly, small steps late, when you're close and want to settle precisely.
Networks deeper than twenty or so layers were, for years, essentially untrainable — every layer's input distribution kept shifting as the layers below it updated, forcing each layer to constantly re-adapt to a moving target. Batch normalisation, and the family of normalisation layers it started, fixed this directly by re-standardising activations at every layer, which is what let networks scale to hundreds of layers.
Every deep learning project, regardless of architecture or task, runs the same twenty-odd lines at its core: forward, compute loss, zero the gradients, backward, step. That order is not negotiable — get it wrong and training either does nothing or does something subtly incorrect, and both failure modes tend to fail quietly rather than crash loudly.
For two decades, networks deeper than a handful of layers simply refused to train — not because the architecture was wrong, but because the gradient signal reaching the earliest layers had either shrunk to numerical zero or grown to numerical infinity by the time it arrived there. Understanding exactly why this happens is what makes the fixes (initialisation, activations, normalisation, residual connections) make sense as one coherent story rather than a grab-bag of tricks.
Before the first gradient is ever computed, a choice has already been made that decides whether training has any chance of working: how the weights start. Get it wrong and the signal either dies (shrinks to zero within a few layers) or explodes (grows without bound) before a single useful gradient update happens.