RLHF and Preference Optimization
Everything in this section so far assumed a reward function already exists. For "write a helpful, harmless response" — the actual goal behind training a modern chat model — no programmable reward function exists at all. RLHF is the answer: learn a reward function from human comparisons, then optimise against it. This is also, for most readers, where reinforcement learning actually shows up in practice.