CNN Architectures
Fifteen years of vision architecture research boiled down to a handful of ideas that survived contact with reality: go deeper, but only once you can actually train the depth; use small filters repeatedly rather than large ones once; and share computation aggressively when compute or memory is scarce. Nearly every architecture below is one of these ideas, applied and refined.
Skip Connections and Depth
In 2015, researchers found something strange: a 56-layer network had higher training error than a 20-layer network on the same task — not overfitting (that would show as a validation gap), but a genuine failure to optimise the deeper network at all. The fix, adding the input back to a layer's output, was a two-line change that took feasible network depth from roughly twenty layers to over a thousand.