Backpropagation
Every parameter in a deep network needs its own gradient, and a network can have millions of them. Backpropagation is the algorithm that computes every single one, applying the chain rule from Calculus and Gradients systematically backwards through the computational graph — at roughly the same cost as one forward pass. That efficiency is the entire reason deep learning became computationally feasible.