Pretraining Objectives
A language model isn't given labelled examples of "good writing" or "correct facts" — it's given unlabelled text and one instruction: predict something that was hidden. Where the knowledge in a modern language model actually comes from traces back to this single trick, repeated over trillions of tokens.
Self-Supervised Vision
Labelled image data is expensive; unlabelled photos are nearly free. Self-supervised vision learns useful visual features from the unlabelled kind alone, by manufacturing a training signal directly from the image itself — the vision-side counterpart to Pretraining Objectives's masked and causal language modelling.