A model can be wrong in exactly two ways: it can be too simple to capture the real pattern (bias), or too sensitive to the particular training sample it happened to see (variance). Diagnosing which one you're facing tells you whether to add capacity or add data — the two most common fixes are not interchangeable.
Training a model is repeated use of one operation: measure the slope of the loss with respect to each parameter, then step in the direction that makes the loss smaller. Every optimiser in this knowledge base — from plain gradient descent to Adam — is a variation on that one move. This page derives the machinery once so later pages can use it without re-deriving it.
Geometric intuitions built in two or three dimensions stop being true once you have two hundred features. As dimension grows, volume concentrates in the corners, distances between points converge toward a single value, and "nearest neighbour" stops meaning much of anything.
Preprocessing decides more of a model's final performance than the choice of algorithm does. It also has to be treated as part of the model, not a one-off step: whatever transformation is applied to training data must travel with the model into production and be fitted only on training data, never on validation or test data (see Train/Validation/Test Splits).
Accuracy is the wrong metric more often than it is the right one. A model that predicts "healthy" for every patient in a dataset where 99% of patients are healthy scores 99% accuracy while being completely useless. Picking the right metric means picking it from the cost of each error type, not from convention.
Regression metrics all try to say "how close were the predictions" in a single number, but they disagree about what "close" means — and that disagreement matters exactly when outliers are present or when comparing models. Knowing which metric lies to you in which situation is the actual skill.
Nearly every model in this knowledge base is trained the same way: compute the gradient of the loss, take a small step in the opposite direction, repeat. The entire difficulty is in choosing how big a step and how much noise to tolerate along the way.
Cross-entropy is the default classification loss for a precise reason: it measures how many extra bits you waste describing reality with the wrong distribution, and minimising it means matching the truth. This page builds entropy, cross-entropy, and KL divergence from scratch so that reason stops being a slogan and becomes a derivation.
Before picking an algorithm, answer one question: what does the training signal look like? A dataset of (input, correct-output) pairs calls for a different family of methods than a pile of unlabelled data, which in turn differs from a system that only gets a delayed reward for a sequence of actions. The paradigm is chosen by the data you have, not by which algorithm sounds most impressive.
A layer in a neural network, a linear regression model, and a batch of predictions being computed all at once are the same operation: multiply a matrix by a vector (or another matrix). Nearly every piece of notation in this knowledge base is linear algebra, so this page fixes the vocabulary once.
The loss is the only thing the model actually optimises. Every other design choice — architecture, optimiser, regularisation — is in service of minimising this one number. Choose it carelessly and the model will optimise exactly what you asked for, which is often not what you meant.
A model that memorises its training set — including its noise and its idiosyncrasies — is worthless the moment it sees a new example. Regularisation is the collection of techniques that stop a model from doing that, by encoding a preference for simpler explanations somewhere in the loss, the data, or the training procedure itself.
A classifier doesn't output "the answer" — it outputs a belief, expressed as a probability distribution over possible answers. Every loss function in this knowledge base is a statement about how that belief compares to reality. This page is the probability vocabulary everything downstream assumes you already have.
Fitting a model is estimating parameters from a finite sample — and every estimate comes with uncertainty about how wrong it might be. This page derives maximum likelihood estimation, the principle underlying nearly every loss function used in this knowledge base, and shows the bridge from "most likely parameters" to "squared error" and "cross-entropy."
Every ML project — a Kaggle competition, a production fraud model, a research paper — runs the same loop. Modelling gets the most attention because it's the most fun to write about, but it is a small slice of the actual work. Problem framing, data quality, and evaluation discipline decide whether the project ever produces something useful; the model is often the easy part.
The test set is spent the moment you make a decision based on it. If you tune a hyperparameter, pick a model, or even decide "let's try one more architecture" after looking at test performance, that number is no longer an honest estimate of how the model will do on truly new data. The validation set exists specifically to absorb those decisions so the test set can stay clean.
Traditional software is a set of rules a person wrote down specify the objective, let optimisation find the rule.
Almost everything in Foundations assumes your observations are independent and identically distributed — that shuffling the rows changes nothing. Time series violates that assumption in the first sentence: the order is the signal. Every technique in this section exists because that one assumption fails.