Value Functions and Bellman Equations
"How good is this state?" sounds unanswerable without simulating every possible future — except it isn't, because of a recursive identity that turns an infinite lookahead into a one-step relationship. That identity, the Bellman equation, is the single mathematical tool every algorithm in this section exploits in some form.