Policy Gradient Methods needed a full Monte Carlo return to compute a low-noise gradient — waiting for episodes to finish, and paying for it in variance. Actor-critic methods replace that Monte Carlo estimate with a second, learned network that predicts value directly, combining the direct optimisation of policy gradients with the low variance of TD learning.
Q-Learning and SARSA hit a hard ceiling: a table indexed by state doesn't scale past small, discrete state spaces. Swap the table for a neural network and the state-space problem disappears — but a new one appears immediately, because naive function approximation in RL is unstable in ways tabular methods never are.
If you already know exactly how the environment works — every transition probability, every reward — you don't need to act, explore, or learn from experience at all. You can compute the optimal policy directly, by repeatedly applying the Bellman equations from Value Functions and Bellman Equations until they converge.
Every algorithm in this section has quietly relied on some exploration mechanism — ε-greedy, entropy bonuses — without asking whether that mechanism is actually a good one. An agent that only ever exploits what it currently believes is best will never discover a better option it hasn't tried; exploration is what makes discovery possible at all.
The Reinforcement Learning Problem described the agent-environment loop informally. The Markov Decision Process (MDP) is the formalism that makes that loop mathematically precise — and precise enough that every algorithm in this section can be stated and analysed against it.
Dynamic Programming needed a fully known model of the environment — real problems rarely offer one. This page introduces the two foundational ways to learn value functions purely from experience: waiting to see how things actually turned out, or updating immediately using your own current guess.
Every algorithm so far learns a value function first, and derives a policy from it only indirectly (act greedily with respect to the values). Policy gradient methods skip the middleman entirely — parameterise the policy directly, and take gradients of expected return with respect to those parameters.
Every algorithm so far has taken a plain gradient step on the policy — and a plain gradient step, if it's too large, can destroy the policy in a way a large supervised-learning update never does. Proximal Policy Optimization is the algorithm most production RL systems actually run, and its entire design is built around preventing exactly that failure.
Monte Carlo and TD Learning only evaluated a fixed policy. This page turns TD prediction into control — algorithms that learn to act well, with no model of the environment — and the two most fundamental such algorithms differ by exactly one term in their update rule.
Everything in this section so far assumed a reward function already exists. For "write a helpful, harmless response" — the actual goal behind training a modern chat model — no programmable reward function exists at all. RLHF is the answer: learn a reward function from human comparisons, then optimise against it. This is also, for most readers, where reinforcement learning actually shows up in practice.
Every model so far in this curriculum has learned from labelled examples — here is the input, here is the correct output. Reinforcement learning removes the labels entirely: an agent takes actions, receives only a scalar signal for how well it did, and has to figure out which of its own past decisions deserve the credit.
"How good is this state?" sounds unanswerable without simulating every possible future — except it isn't, because of a recursive identity that turns an infinite lookahead into a one-step relationship. That identity, the Bellman equation, is the single mathematical tool every algorithm in this section exploits in some form.