The cheapest way to get more training data isn't collecting more — it's transforming the data you already have in ways that shouldn't change the label. A flipped photo of a cat is still a cat; a rotated photo of a stop sign is still a stop sign (probably). What you choose to augment with is a direct statement of what invariances you want the model to learn.
A model that memorises its training set — including its noise and its idiosyncrasies — is worthless the moment it sees a new example. Regularisation is the collection of techniques that stop a model from doing that, by encoding a preference for simpler explanations somewhere in the loss, the data, or the training procedure itself.
A modern network routinely has more parameters than training examples — by classical statistical intuition, this should guarantee catastrophic overfitting. It usually doesn't, and the regularisation techniques on this page are less about shrinking weights (as in Overfitting and Regularization's classical L1/L2 story) and more about injecting noise or stopping early.
Unregularised least squares fits every quirk of the training sample, including its noise, once you have enough features relative to examples. Ridge, lasso, and elastic net apply the general regularisation principle from Overfitting and Regularization specifically to linear models, each trading a little bias for a large drop in variance.