DQN CartPole — Neural Network Learns to Balance
Watch a neural network learn to balance a pole on a cart — right in your browser! This is Deep Q-Network (DQN) with TensorFlow.js: experience replay, target network, and ε-greedy exploration. See the AI go from random flailing to 500-step perfect balance in real-time. 100% browser-side, no server.
⚙️ Hyperparameters
Episode Reward (higher = better, 500 = solved)
Training Loss
DQN: When Q-Tables Can't Hold the World
1. The Curse of Continuous States
In a maze, there are 81 discrete cells — a Q-table works fine. But CartPole has continuous states: position, velocity, angle, angular velocity. There are infinitely many states. You can't store Q(s,a) in a table. You need a function approximator.
2. Neural Network as Q Function
DQN uses a neural network to approximate Q(s,a). Input: the 4-dimensional state. Output: Q-values for each action. The network learns to generalize — similar states get similar Q-values, even if never visited before. This is the key to handling continuous spaces.
3. Experience Replay
Consecutive experiences are highly correlated (state s leads to s'). Training on correlated data makes neural networks diverge. DQN stores experiences in a replay buffer and samples random mini-batches. This breaks correlation and reuses each experience multiple times.
4. Target Network
The target Q-value uses the network's own output: y = r + γ max Q(s',a'). If we update Q, the target also changes — like chasing a moving target. DQN uses a separate target network (a frozen copy) to compute targets. Updated every N steps for stability.
5. ε-Greedy Exploration
Early on, the network's Q-values are garbage. Following them would be disastrous. ε-greedy: with probability ε, take a random action; otherwise, follow the network. ε starts at 1.0 (pure random) and decays to 0.01, gradually shifting from exploration to exploitation.
6. The "Solved" Threshold
CartPole-v1 is considered "solved" when the average reward over 100 consecutive episodes reaches 475 (out of 500 max). With good hyperparameters, DQN typically solves it in 50-200 episodes. Watch the reward chart climb from ~20 (random) to 500 (perfect).
Frequently Asked Questions
What is DQN (Deep Q-Network)?
DQN extends Q-Learning to continuous state spaces by replacing the Q-table with a neural network. Instead of looking up Q(s,a) in a table, the network predicts Q(s,a) from the state vector. This allows the agent to handle infinite state spaces like CartPole's continuous physics. DQN was introduced by DeepMind in 2013 and famously learned to play Atari games at human level.
Why does DQN need Experience Replay?
Neural networks assume their training data is i.i.d. (independent and identically distributed). But RL experiences are sequential and highly correlated — state st directly leads to st+1. Training on sequential data causes the network to overfit to recent experiences and forget older ones. The replay buffer stores past experiences and samples them randomly, breaking the correlation and making training stable.
Why does DQN need a Target Network?
In Q-Learning, the target is y = r + γ·maxa' Q(s',a'). Both the prediction and the target use the same Q function. When you update Q to better predict the target, the target itself shifts — creating a feedback loop that causes divergence. The target network is a frozen copy of Q, updated every N steps. By using it to compute targets, the target stays stable between updates, breaking the feedback loop.
How long does it take to train?
In your browser, DQN typically solves CartPole in 50-200 episodes (about 1-5 minutes depending on your CPU and speed setting). On a GPU it would be faster, but the bottleneck is actually the environment stepping (JavaScript), not the neural network forward/backward pass. Increasing "Speed" to 10-20 steps/frame makes training much faster but less visual.
What happens if I change the hidden units?
More hidden units = more capacity = can learn more complex patterns, but slower per step and more prone to overfitting. For CartPole (a simple 4-state problem), 64-128 units is plenty. Too few (like 16) may not have enough capacity. Too many (like 256) wastes computation. The default 64 is a good starting point.
Is this the same DQN that plays Atari games?
Same algorithm, different scale. DeepMind's Atari DQN used a convolutional neural network (CNN) to process raw pixel images (84×84×4 frames) as state. Our DQN uses a simple MLP (fully-connected layers) with the 4 physics values as state. The core algorithm — experience replay, target network, ε-greedy — is identical. The architecture difference reflects the input: images need CNNs, physics values don't.
Why does it sometimes "forget"?
This is called catastrophic forgetting. When the agent explores new states, the network updates to handle them, potentially degrading performance on previously mastered states. Experience replay mitigates this by keeping old experiences in the buffer, but it doesn't fully solve it. If you see the reward chart dip after reaching high values, that's forgetting. Techniques like prioritized experience replay (PER) help further.