Optimizers
Plain gradient descent takes the same fixed-size step in every direction, every time, regardless of the loss surface's shape. Everything covered on this page is a way of using information from past gradients to take smarter steps — and by the mid-2010s, one method (Adam) had absorbed most of these ideas and become the default nearly everyone reaches for first.