REINFORCE — Policy Gradient Learns from Full Episodes
Watch a neural network learn a policy directly — no Q-values, no value function! REINFORCE (Monte Carlo Policy Gradient) is the purest form of policy-based RL. The agent plays full episodes, then updates its strategy using the policy gradient theorem: ∇J = E[∇log π(a|s) · G]. See action probabilities shift in real-time as the AI learns. TensorFlow.js, 100% browser-side.
Action Probabilities π(a|s) — Live Policy
Unlike DQN which outputs Q-values, REINFORCE outputs action probabilities. Watch the bars shift as the policy learns.
Policy Landscape π(right | θ, θ̇)
Action probability across the state space (pole angle × angular velocity). Blue = push right, Red = push left. This visualization is unique to policy-based methods.
⚙️ Hyperparameters
Episode Reward (500 = solved)
Episode Returns (Monte Carlo G — shows variance)
REINFORCE's weakness: high variance. The wider the spread, the noisier the gradient.
REINFORCE: The Purest Policy Gradient
1. Value-based vs Policy-based
DQN learns Q(s,a) — the value of each action — then picks the best. REINFORCE skips the middleman: it directly learns π(a|s) — the probability of choosing each action. No max operation, no Q-table, just a direct mapping from states to action probabilities.
2. The Policy Gradient Theorem
The core equation: ∇J(θ) = E[∇log π(a|s) · G]. Here G is the Monte Carlo return (discounted sum of future rewards). If an action led to high returns, the gradient pushes π(a|s) up; if low returns, it pushes down. The log comes from the score function trick — it converts "change probability" into "change network weights".
3. Monte Carlo: Wait for the Full Episode
Unlike DQN (which updates every step using TD bootstrapping), REINFORCE must wait until the episode ends. Only then does it know the actual return G for each step. This is "Monte Carlo" — you need the complete trajectory. The trade-off: no bootstrapping bias, but high variance and slower updates.
4. The Variance Problem
Returns G can range from 0 to 500 in CartPole. A single lucky episode can produce huge gradients, while an unlucky one produces tiny ones. This variance makes training unstable. The "Returns σ" stat shows this — high σ means noisy gradients. This is why the next algorithm (A2C) introduces a baseline.
5. On-Policy: No Experience Replay
DQN has an experience replay buffer — it stores old transitions and replays them. REINFORCE cannot do this. The policy gradient requires data collected by the current policy. Old episodes were collected by a different policy, so their gradients point in the wrong direction. This makes REINFORCE sample-inefficient: every episode is used once, then discarded.
6. Why Softmax Output?
The policy network ends with a softmax layer, outputting valid probabilities that sum to 1. This ensures the policy is a proper probability distribution. During training, the softmaxCrossEntropy loss naturally computes -log π(a|s), which is exactly the score function needed for the policy gradient.
Frequently Asked Questions
What is the difference between REINFORCE and DQN?
DQN is value-based: it learns Q(s,a) and derives a policy by picking argmax. REINFORCE is policy-based: it directly learns π(a|s) as action probabilities. DQN uses TD learning (updates every step), experience replay (reuses old data), and a target network. REINFORCE uses Monte Carlo (updates after full episode), no replay buffer (on-policy only), and no target network. DQN is off-policy; REINFORCE is on-policy.
What does ∇log π(a|s) mean?
This is the "score function" — the gradient of the log-probability of action a with respect to network parameters θ. It tells us: "if I want to increase the probability of action a, which direction should I move the weights?" Multiplying by G (the return) scales this gradient: good actions (high G) get pushed up, bad actions (low G) get pushed down. The expectation E[·] averages over all possible trajectories.
Why does REINFORCE have high variance?
The return G is a sum of random rewards over a full episode. In CartPole, G can be anywhere from 1 (immediate failure) to 500 (perfect balance). This 500× range means the gradient magnitude varies wildly between episodes. One good episode produces a huge gradient update; one bad episode produces a tiny one. This makes training noisy and slow. A2C fixes this by subtracting a baseline V(s) — instead of using raw G, it uses the advantage A = G - V(s), which has much lower variance.
Why can't REINFORCE use experience replay?
The policy gradient ∇J = E[∇log π(a|s) · G] assumes the data comes from the current policy π. Old episodes were collected by a different policy π_old. Using them would compute gradients for π_old, not π — the update would be wrong. This is the fundamental meaning of "on-policy": the data collector and the learner must be the same policy. DQN doesn't have this constraint because Q-learning is off-policy — the Bellman equation holds regardless of which policy collected the data.
What is return normalization?
Instead of using raw returns G, we standardize them: subtract the mean and divide by the standard deviation. This makes the gradient magnitudes consistent across episodes, regardless of whether it was a good or bad episode. It doesn't change the optimal direction (just scales it), but dramatically stabilizes training. It's a practical trick not always mentioned in textbooks.
What is the policy landscape heatmap?
It shows π(right | θ, θ̇) — the probability of pushing right — for every combination of pole angle (x-axis) and angular velocity (y-axis). Blue means "push right", red means "push left". As the policy learns, you'll see clear regions form: when the pole leans right (positive angle), the policy should push right (blue); when it leans left, push left (red). This visualization is unique to policy-based methods — DQN can only show Q-values, not direct action probabilities.
Is REINFORCE used in practice?
Rarely on its own — the variance is too high for complex problems. But REINFORCE is the foundation of all policy gradient methods. A2C adds a critic (baseline). PPO adds trust region clipping. SAC adds entropy regularization. All of them trace back to the same ∇log π(a|s) · advantage structure. Understanding REINFORCE is prerequisite to understanding every modern policy-based algorithm.