2 docs tagged with "calculus"

Every parameter in a deep network needs its own gradient, and a network can have millions of them. Backpropagation is the algorithm that computes every single one, applying the chain rule from Calculus and Gradients systematically backwards through the computational graph — at roughly the same cost as one forward pass. That efficiency is the entire reason deep learning became computationally feasible.

Training a model is repeated use of one operation: measure the slope of the loss with respect to each parameter, then step in the direction that makes the loss smaller. Every optimiser in this knowledge base — from plain gradient descent to Adam — is a variation on that one move. This page derives the machinery once so later pages can use it without re-deriving it.