RL Benchmark — Compare All Algorithms Side by Side
The grand finale: run DQN, Double DQN, REINFORCE, A2C, and PPO simultaneously on the same CartPole environment and watch their learning curves diverge in real time. Which algorithm converges fastest? Which is most stable? Which solves CartPole in the fewest episodes? See the answer unfold before your eyes. Plus a complete algorithm selection decision tree — when to use value-based vs policy-based, on-policy vs off-policy, discrete vs continuous. TensorFlow.js, 100% browser-side.
Select Algorithms to Compare (2–5)
Episode Reward — Comparative Learning Curves
Smoothed Avg (10-episode window)
Which Algorithm Should You Choose?
1. Value-Based vs Policy-Based
Value-based (DQN family) learn Q(s,a) and derive policy indirectly via argmax. Great for discrete actions, sample-efficient (off-policy replay). Policy-based (REINFORCE, PPO) learn π(a|s) directly. Essential for continuous actions, can learn stochastic policies, but typically need more data.
2. On-Policy vs Off-Policy
Off-policy (DQN, SAC) can reuse old data via replay buffer — more sample-efficient. On-policy (REINFORCE, A2C, PPO) must collect fresh data after each update — less efficient but more stable. PPO's clip gives it a "semi-on-policy" middle ground: reuse data for K epochs, then discard.
3. Discrete vs Continuous Actions
Discrete (left/right, up/down): DQN family works. Just output Q-values per action and pick max. Continuous (torque, angle): max_a Q(s,a) becomes an optimization problem with no closed form. You need policy-based methods (PPO, SAC, DDPG). SAC is the gold standard for continuous control.
4. Stability vs Speed
REINFORCE is simple but high-variance — training can collapse. A2C adds a critic to reduce variance. PPO adds clipping for extra stability and data reuse. DQN is stable but needs careful tuning (target network, replay buffer). Double DQN fixes overestimation. In practice: PPO for reliability, SAC for continuous, DQN for discrete simplicity.
5. Sample Efficiency Ranking
How many environment steps to solve CartPole? Roughly: SAC > Double DQN > DQN > PPO > A2C > REINFORCE. Off-policy methods (DQN, SAC) reuse data via replay, so they learn more per step. On-policy methods (PPO, A2C, REINFORCE) must collect fresh data each update. But sample efficiency isn't everything — wall-clock time also depends on compute per step.
6. The Decision Tree
Continuous actions? → SAC (or PPO if you need simplicity). Discrete actions + simple environment? → DQN or Double DQN. Discrete + need stability/reliability? → PPO. Just learning? → Start with REINFORCE (simplest policy gradient), then A2C. Production RL system? → PPO (OpenAI's default) or SAC (for control tasks).
Algorithm Cheat Sheet
| Algorithm | Type | Policy | Actions | Replay Buffer | Sample Eff. | Stability | Best For |
|---|---|---|---|---|---|---|---|
| ● DQN | Value-based | Off-policy | Discrete | ✓ | Medium | Medium | Simple discrete tasks |
| ● Double DQN | Value-based | Off-policy | Discrete | ✓ | Medium | Medium-High | Discrete tasks (less overestimation) |
| ● REINFORCE | Policy Gradient | On-policy | Both | ✗ | Low | Low | Learning the basics |
| ● A2C | Actor-Critic | On-policy | Both | ✗ | Low-Med | Medium | Simple policy-based control |
| ● PPO | Actor-Critic | On-policy* | Both | ✗ | Medium | High | Production RL (RLHF, robotics) |
| ● SAC | Actor-Critic | Off-policy | Continuous | ✓ | High | High | Continuous control (robots) |
* PPO is technically on-policy but reuses data for K epochs via clipping — a "semi-on-policy" middle ground.
Frequently Asked Questions
Why can't DQN handle continuous actions?
DQN's core operation is max_a Q(s,a) — find the action with the highest Q-value. With discrete actions (e.g., left/right), you just compare 2 values. With continuous actions (e.g., torque in [-2, 2]), there are infinitely many actions — you can't enumerate them. You'd need to solve an optimization problem at every step. Policy-based methods (PPO, SAC) sidestep this entirely: the policy π(a|s) directly outputs the action, no maximization needed.
Which algorithm is the "best"?
There is no universal best — it depends on your problem. For discrete actions and simplicity: Double DQN. For production reliability: PPO (OpenAI's default, used in RLHF for ChatGPT). For continuous control: SAC (maximum entropy framework, best sample efficiency). For learning: start with REINFORCE to understand policy gradients, then A2C to see how a critic helps, then PPO to see how clipping stabilizes training.
Why does REINFORCE perform worse than the others?
REINFORCE uses pure Monte Carlo returns with no baseline. The gradient ∇log π(a|s) · G has enormous variance because G (the episode return) can swing wildly between episodes. A2C reduces this by subtracting a learned baseline V(s): ∇log π(a|s) · (G - V(s)). PPO goes further with clipping and multiple epochs. In our benchmark, you'll typically see REINFORCE learn slower and less stably than A2C and PPO.
What is "sample efficiency"?
Sample efficiency measures how much the agent learns per environment step. Off-policy algorithms (DQN, SAC) are more sample-efficient because they store experiences in a replay buffer and reuse them. On-policy algorithms (REINFORCE, A2C, PPO) must discard data after each update. In our benchmark, DQN and Double DQN typically need fewer episodes to solve CartPole than REINFORCE or A2C. PPO is in between — it reuses data for K epochs before discarding.
Why are the results different each time I run?
RL training is stochastic: random network initialization, random exploration (ε-greedy or stochastic policy), and random experience sampling all introduce variance. The same algorithm can solve CartPole in 50 episodes on one run and 200 on another. This is why we show learning curves rather than single numbers — the trend matters more than any individual episode. Run the benchmark multiple times to get a feel for the variance.
How do these algorithms compare to the ones used in industry?
These are the same algorithms used in production, just at smaller scale. ChatGPT's RLHF uses PPO (our Episode 10). Boston Dynamics-style robot control uses SAC or PPO (our Episode 11). DeepMind's Atari-playing AI used DQN (our Episode 6) and its descendants. The core math is identical — the difference is scale: larger networks, GPU compute, distributed training. But the algorithm is the same.
What about algorithms not in this series?
We covered the most important ones, but there are many others. DDPG and TD3 are deterministic policy gradient methods for continuous control (SAC is generally better). TRPO is PPO's predecessor (more theoretically grounded but harder to implement). Rainbow DQN combines 6 DQN improvements. AlphaZero uses Monte Carlo Tree Search + deep RL for board games. Dreamer learns a world model and plans in it. If you understand the 11 algorithms in this series, you have the foundation to learn all of these.