Double Q-Learning โ Overestimation Bias Visualizer
See why Q-Learning overestimates Q-values and how Double Q-Learning fixes it! Watch Q-Learning, Double Q-Learning, and the exact optimal Q* (via Value Iteration) learn side-by-side on Cliff Walking. Real-time bias chart shows Q-Learning's systematic positive bias vs Double Q-Learning's unbiased estimates. 100% browser-side.
โ๏ธ Hyperparameters (shared)
Q-Value Bias Over Training
How much each algorithm overestimates Q* โ lower (closer to 0) is better. Q-Learning shows systematic positive bias; Double Q-Learning stays near zero.
Double Q-Learning: Killing the Overestimation Bias
1. The Problem
Q-Learning uses max Q(s',a') in its update. In noisy environments, max is positively biased — it picks the highest value, which is likely inflated by random noise. This causes systematic overestimation of Q-values.
2. The Insight
If you use the same Q-table to both select and evaluate the best action, any noise gets amplified. But if you use two independent tables — one to select, one to evaluate — the noise cancels out on average.
3. Double Q-Learning
Maintain two tables QA and QB. Each step, randomly update one using the other: Q_A(s,a) โ Q_A + ฮฑ[r + ฮณยทQ_B(s', argmax Q_A(s',ยท)) โ Q_A]. The decoupling breaks the bias.
4. Value Iteration (Ground Truth)
We solve the environment exactly using dynamic programming (Bellman optimality), giving us the true optimal Q*. This lets us measure how much each algorithm overestimates.
5. Watch the Bias
The bottom chart tracks the average overestimation over training. Q-Learning's bias stays positive (it thinks states are better than they are). Double Q-Learning's bias hovers near zero — it's unbiased by design.
6. Does It Matter?
For this small grid, the bias doesn't change the policy much. But in complex environments (like DQN on Atari), overestimation causes the agent to fixate on overvalued actions, leading to suboptimal policies. Double DQN was a key DQN improvement.
Frequently Asked Questions
What is the overestimation bias in Q-Learning?
Q-Learning's update uses max Q(s',a'). When Q-values contain noise (which they always do during learning), the max operation systematically picks the most optimistically overestimated value. This is a positive bias — Q-Learning tends to think states are better than they actually are. The bias is mathematically provable: E[max(X)] โฅ max(E[X]) for any random variable X.
How does Double Q-Learning fix this?
It maintains two independent Q-tables, QA and QB. On each step, it randomly picks one to update (say QA). It uses QA to select the best next action (argmax Q_A(s',ยท)), but uses QB to evaluate that action (Q_B(s', a*)). Because QA and QB are updated with different experiences, their noise is independent. The expected value of QB at the action chosen by QA is unbiased.
Why is Value Iteration shown as "ground truth"?
Value Iteration is a model-based method: it knows the exact transition probabilities and rewards, and computes the optimal Q* by iteratively applying the Bellman optimality equation until convergence. Since we defined the Cliff Walking environment, we know its dynamics exactly. Value Iteration gives us the true optimal Q-values, which we use as a reference to measure how much Q-Learning and Double Q-Learning deviate.
If we know the environment, why use RL at all?
Great question! In real-world RL, we usually don't know the environment dynamics — that's why we use model-free RL. But for educational purposes, using a known environment lets us compute the ground truth and visualize the bias. This is a luxury we have in teaching that we don't have in practice.
Does Double Q-Learning always learn better policies?
Not necessarily. In simple environments like this grid, both Q-Learning and Double Q-Learning converge to similar (optimal) policies — the bias affects Q-values but not always the resulting policy. The difference becomes critical in complex environments with large state spaces (like DQN on Atari games), where overestimation causes the agent to persistently choose overvalued actions, leading to poor exploration and suboptimal policies.
What is the relationship between Double Q-Learning and Double DQN?
Double DQN (Van Hasselt et al., 2016) is the deep learning version of Double Q-Learning. In DQN, the target network and online network naturally provide two "independent" estimators. Double DQN uses the online network to select actions and the target network to evaluate them — exactly the same decoupling principle as Double Q-Learning, but with neural networks instead of tables.
Is this really running in my browser?
Yes! Q-Learning, Double Q-Learning, and Value Iteration all run in pure JavaScript on your device. No server, no API calls, no data upload. Value Iteration runs once at initialization (it converges in milliseconds for this small grid).