A2C — Actor-Critic with Advantage Baseline
REINFORCE has high variance. A2C fixes it by adding a Critic network that estimates V(s) as a baseline. The Actor now learns from the advantage A = G - V(s) — "how much better was this action than expected?" Watch two agents train side by side: vanilla REINFORCE vs A2C with baseline. See the variance drop and training stabilize. TensorFlow.js, 100% browser-side.
⚙️ Hyperparameters (shared by both agents)
Episode Reward (higher = better)
Gradient Variance (lower = more stable)
A2C's advantage A = G − V(s) should have much lower variance than REINFORCE's raw G.
A2C: Adding a Critic to Tame REINFORCE's Variance
1. The Variance Problem
REINFORCE uses ∇log π(a|s) · G where G ranges from 1 to 500. A lucky episode (G=400) produces a huge gradient; an unlucky one (G=20) produces a tiny one. This 20× range makes training noisy. The agent can't tell if an action was "good for this state" or just "happened to be in a good episode".
2. The Baseline Idea
Instead of asking "was this action good?" (absolute), ask "was this action better than expected?" (relative). We need a reference point: the expected return from state s, denoted V(s). If G > V(s), the action was better than average; if G < V(s), it was worse. This relative measure is the advantage A = G − V(s).
3. The Critic Network
V(s) is unknown — we need to learn it. Enter the Critic: a second neural network that takes state s and outputs a single value V(s). The Critic is trained with MSE loss: target = G (the actual return). As the Critic learns, V(s) converges to the true expected return from each state.
4. The Actor-Critic Dance
The Actor (policy network) learns from advantage: ∇log π(a|s) · A. The Critic (value network) learns from returns: (V(s) − G)². They train simultaneously: Critic gets better at predicting V(s), which gives Actor a better baseline, which reduces Actor's variance, which makes Actor learn faster — a virtuous cycle.
5. Why Variance Drops
Var(A) = Var(G − V) = Var(G) + Var(V) − 2·Cov(G,V). Since V(s) is trained to predict G, Cov(G,V) ≈ Var(V). So Var(A) ≈ Var(G) − Var(V). The better the Critic, the more variance is "explained" by V(s) and subtracted out. In practice, variance can drop 5-10×.
6. Why Not Just Normalize Returns?
Return normalization (from our REINFORCE tool) subtracts the batch mean — a single number for all steps. The Critic subtracts a state-specific V(s) — different baseline for each state. This is much more precise: a return of 100 might be great from a bad state but terrible from a good state. The Critic knows the difference; batch normalization doesn't.
Frequently Asked Questions
What is the advantage function?
The advantage A(s,a) = Q(s,a) − V(s) measures how much better action a is compared to the average action in state s. In REINFORCE (no Q-network), we approximate it as A ≈ G − V(s), where G is the Monte Carlo return. If A > 0, the action was better than expected; if A < 0, it was worse. The policy gradient becomes ∇log π(a|s) · A instead of ∇log π(a|s) · G.
Why does the baseline not bias the gradient?
This is a key theoretical result: subtracting a state-dependent baseline V(s) does not change the expected gradient. Mathematically, E[∇log π(a|s) · V(s)] = 0 because V(s) doesn't depend on the action a (it's only a function of s). The baseline only affects the variance, not the mean. This is why we can safely subtract V(s) — the gradient still points in the correct direction on average, just with less noise.
What is the difference between Actor and Critic?
The Actor is the policy network π(a|s) — it decides which actions to take. It's trained with the policy gradient using the advantage as weight. The Critic is the value network V(s) — it estimates how good a state is. It's trained with MSE regression against actual returns. The Actor "acts", the Critic "judges". Together they form the Actor-Critic architecture, the foundation of A2C, A3C, PPO, and SAC.
Why is the Critic learning rate higher than the Actor's?
In our tool, the Critic LR is 2× the Actor LR (0.02 vs 0.01). This is a common practice: the Critic needs to converge before the Actor can benefit from an accurate baseline. If the Critic lags behind, the advantage estimates are noisy, and the Actor's training is unstable. By giving the Critic a higher LR, it "leads" the Actor — the baseline improves faster than the policy changes.
Is A2C on-policy or off-policy?
A2C is on-policy, just like REINFORCE. The data must be collected by the current policy. The Critic doesn't change this — it only provides a baseline, it doesn't enable off-policy learning. For off-policy actor-critic, you'd need algorithms like ACER or Q-prop, which use importance sampling corrections.
What is the difference between A2C and A3C?
A3C (Asynchronous Advantage Actor-Critic) runs multiple agents in parallel on different environment instances, each with their own local network, asynchronously updating a global network. A2C (Advantage Actor-Critic) is the synchronous version: one agent, one network, no parallelism. A2C is simpler and often performs just as well. In practice, modern implementations use A2C because GPU parallelism makes async CPU parallelism less necessary.
Can A2C use TD instead of Monte Carlo?
Yes! Instead of waiting for the full episode return G, A2C can use the TD advantage: A = r + γ·V(s') − V(s). This allows updating every step (no need to wait for episode end). However, TD introduces bias (bootstrapping from an imperfect V). Our tool uses Monte Carlo (A = G − V) to keep the comparison with REINFORCE fair — both wait for full episodes. TD-based A2C is covered conceptually in the blog post.