Skip Connections and Depth
In 2015, researchers found something strange: a 56-layer network had higher training error than a 20-layer network on the same task — not overfitting (that would show as a validation gap), but a genuine failure to optimise the deeper network at all. The fix, adding the input back to a layer's output, was a two-line change that took feasible network depth from roughly twenty layers to over a thousand.