Fraud, equipment failure, network intrusions — the interesting class is often so rare that a supervised classifier never sees enough positive examples to learn from. Anomaly detection reframes the problem entirely: instead of learning what the rare class looks like, learn what normal looks like, and flag anything that deviates.
This page uses xgboost, lightgbm, and catboost — none is in the knowledge base's default library set (numpy, scipy, pandas, matplotlib, scikit-learn, torch, torchvision, transformers, datasets). They are the industry-standard implementations of Gradient Boosting and this page cannot honestly cover the topic without them. Install with pip install xgboost lightgbm catboost.
Every other model in this section requires some statistical literacy to interpret. A decision tree doesn't — you can hand the diagram to someone with no ML background and they can trace a prediction themselves, one yes/no question at a time. That transparency comes at a real cost: trees are greedy, and greedy is not the same as optimal.
Bagging and boosting are both ensembles of the same base learner. This page covers combining genuinely different models — a linear model, a tree ensemble, and a neural network, say — which only helps to the extent those models fail in different ways.
k-means forces every point into exactly one cluster, with no notion of confidence. A Gaussian mixture model asks a softer question: what's the probability this point belongs to each cluster? Answering it requires an algorithm — expectation-maximisation — that recurs throughout probabilistic ML wherever there are hidden variables to infer.
Random forests average many independent trees to cancel out variance. Gradient boosting does something structurally different: it builds trees one at a time, each new tree specifically targeting the mistakes the ensemble has made so far. Where bagging reduces variance, boosting reduces bias — and the combination of the two ideas covers most of what wins tabular ML competitions.
k-means needs to be told $k$ in advance, and assumes every cluster is roughly round. This page covers two families that relax those assumptions in different directions: hierarchical clustering defers the choice of cluster count to a visual cut, and density-based clustering abandons the "round cluster" assumption entirely by defining a cluster as a connected region of high density.
99% of your rows are one class, and accuracy just became a lie. Imbalance is best understood as a metric-and-threshold problem before it's a sampling problem — fixing evaluation costs nothing and should always come first, and often it turns out to be the only fix actually needed.
k-means is the clustering algorithm everyone reaches for first — and its simplicity conceals just how strong its assumptions are. It assumes clusters are round, similarly sized, and similarly dense, and it will happily produce a confident, wrong answer when those assumptions don't hold.
Every model so far has fit parameters during training and thrown the raw data away afterward. kNN does the opposite: it stores the entire training set verbatim and defers all the actual work to inference time, where it asks "which stored examples look most like this new one?"
Some data simply cannot be separated by a straight line — two concentric circles of different classes have no linear boundary at all. The kernel trick lets a linear algorithm operate as if the data had been lifted into a much higher-dimensional (sometimes infinite-dimensional) space where a linear boundary does exist, without ever actually computing that lift.
Every model in this knowledge base is measured against linear regression, and for good reason: it's the only widely-used model whose optimum you can write down in one line, no iteration required. Understanding exactly why that's possible — and exactly when it stops being possible — is the fastest way to understand the rest of classical ML.
Despite the name, logistic regression is a classifier, and it remains the first model worth trying on any new tabular classification task — fast to train, easy to interpret, and a strong baseline against which everything fancier should be measured.
t-SNE and UMAP produce the 2-D scatter plots of high-dimensional embeddings that appear everywhere — clusters of colourful dots, each supposedly a meaningful group. They are genuinely useful, and also genuinely easy to misread: knowing exactly what these plots do and do not preserve is the difference between a real insight and a confident misinterpretation.
Every hyperparameter search is a way of spending a limited resource: the information in your validation set. Search too aggressively, over too many combinations, and you'll quietly overfit to the validation set itself — the exact failure the validation set was supposed to prevent in the first place.
Naive Bayes assumes something that is almost never true — that every feature is independent given the class — and yet remains a strong baseline for text classification decades after more sophisticated methods appeared. Understanding why a false assumption still produces a useful classifier is the real lesson of this page.
Principal component analysis answers a simple question with surprisingly deep machinery: which few directions in a high-dimensional dataset capture most of what's actually going on? The answer — rotate onto the axes of greatest variance — turns out to be one of the most reused mathematical results in all of machine learning.
A single unconstrained decision tree overfits badly. Grow a few hundred of them, each on a slightly different random sample of the data, and average their predictions — and the overfitting largely cancels out. That's the entire idea behind bagging, and random forests are bagging applied specifically to trees with one extra trick.
Unregularised least squares fits every quirk of the training sample, including its noise, once you have enough features relative to examples. Ridge, lasso, and elastic net apply the general regularisation principle from Overfitting and Regularization specifically to linear models, each trading a little bias for a large drop in variance.
Logistic regression finds a separating line. Support vector machines find the separating line furthest from every training point — and it turns out only a handful of points (the "support vectors") actually determine where that line goes. The rest of the training set could be deleted without changing the model at all.