Every model in this knowledge base is measured against linear regression, and for good reason: it's the only widely-used model whose optimum you can write down in one line, no iteration required. Understanding exactly why that's possible — and exactly when it stops being possible — is the fastest way to understand the rest of classical ML.
Despite the name, logistic regression is a classifier, and it remains the first model worth trying on any new tabular classification task — fast to train, easy to interpret, and a strong baseline against which everything fancier should be measured.
Unregularised least squares fits every quirk of the training sample, including its noise, once you have enough features relative to examples. Ridge, lasso, and elastic net apply the general regularisation principle from Overfitting and Regularization specifically to linear models, each trading a little bias for a large drop in variance.