Activation Functions
The non-linearity between layers is what makes depth meaningful at all — From Perceptron to MLP proved that a stack of purely linear layers collapses into a single linear layer. But not all non-linearities are equal: the shape of an activation's derivative determines whether gradients survive a deep network or vanish before reaching the early layers.