Loss Functions
The loss is the only thing the model actually optimises. Every other design choice — architecture, optimiser, regularisation — is in service of minimising this one number. Choose it carelessly and the model will optimise exactly what you asked for, which is often not what you meant.