Skip to main content

3 docs tagged with "ensembles"

View all tags

Ensembles and Stacking

Bagging and boosting are both ensembles of the same base learner. This page covers combining genuinely different models — a linear model, a tree ensemble, and a neural network, say — which only helps to the extent those models fail in different ways.

Gradient Boosting

Random forests average many independent trees to cancel out variance. Gradient boosting does something structurally different: it builds trees one at a time, each new tree specifically targeting the mistakes the ensemble has made so far. Where bagging reduces variance, boosting reduces bias — and the combination of the two ideas covers most of what wins tabular ML competitions.

Random Forests and Bagging

A single unconstrained decision tree overfits badly. Grow a few hundred of them, each on a slightly different random sample of the data, and average their predictions — and the overfitting largely cancels out. That's the entire idea behind bagging, and random forests are bagging applied specifically to trees with one extra trick.