Pretraining Objectives
A language model isn't given labelled examples of "good writing" or "correct facts" — it's given unlabelled text and one instruction: predict something that was hidden. Where the knowledge in a modern language model actually comes from traces back to this single trick, repeated over trillions of tokens.