Dyna-Q โ Planning vs Direct RL Maze Race
Watch Dyna-Q learn a maze 10x faster than Q-Learning using "imagined" experience! After each real step, Dyna-Q replays N planning steps through a learned model. See the shortcut maze experiment: when the environment changes mid-training, Dyna-Q adapts in episodes, not dozens. 100% browser-side.
โ๏ธ Hyperparameters (shared)
Steps Per Episode (lower = better)
Dyna-Q converges faster thanks to planning. After the shortcut opens, watch who adapts first.
Dyna-Q: Learning + Planning = 10x Faster
1. The Problem
Q-Learning only learns from real experience — each step in the environment produces one Q-update. In large environments, you need thousands of real steps to learn a good policy. Real experience is expensive.
2. The Dyna Insight
What if you could replay past experiences? Dyna-Q learns a simple model of the environment (just remembering "what happened when I did X in state Y"), then uses it to imagine extra training — no real interaction needed.
3. The Model
The model is trivially simple: Model(s,a) → (s', r). Just remember the next state and reward for each (s,a) you've experienced. No probabilities, no math — just a lookup table. In deterministic environments, it's perfect.
4. Planning Steps
After each real step, Dyna-Q does N planning steps: randomly sample a previously seen (s,a), look up the model's predicted (s', r), and do a Q-Learning update. This is "free" learning — no real interaction needed. N=50 means 51 updates per real step.
5. The Shortcut Maze
After 30 episodes, a wall opens creating a shortcut. Q-Learning slowly discovers it through random exploration. Dyna-Q's planning propagates the new information rapidly through its model — it adapts in a few episodes, not dozens.
6. Real vs Imagined
Watch the "Real Steps" counter. Dyna-Q reaches optimal performance with 10x fewer real steps than Q-Learning. The planning steps are computationally cheap (just table lookups) but dramatically reduce the need for expensive real interaction.
Frequently Asked Questions
What is the difference between Dyna-Q and Q-Learning?
Q-Learning does one Q-update per real step in the environment. Dyna-Q does the same real-step update, plus N additional "planning" updates using a learned model. Both use the same Q-Learning update rule — the only difference is that Dyna-Q gets N extra updates per step for free. With N=50, Dyna-Q does 51 updates per real step vs Q-Learning's 1.
What is the "model" in Dyna-Q?
The model is just a lookup table: for each (state, action) pair you've experienced, remember the resulting next state and reward. Model[(s,a)] = (s', r). In deterministic environments (like this maze), the model is perfect — it always predicts the correct outcome. In stochastic environments, you'd need to average or sample.
What are "planning" steps?
After each real interaction with the environment, Dyna-Q performs N planning steps. In each planning step, it randomly picks a previously visited (s,a), looks up what happened next in its model, and does a Q-Learning update as if it were real. These updates are "free" — they don't require any real interaction, just memory and computation.
Why does the shortcut maze matter?
It demonstrates planning's power in adaptation. When the environment changes (a wall opens), Q-Learning must stumble upon the new path through exploration. Dyna-Q's planning rapidly propagates the new information: once it discovers the shortcut once, planning steps spread that knowledge to neighboring states in the model, so it can exploit the new path almost immediately.
Is Dyna-Q model-based or model-free?
It's both! Dyna-Q is a hybrid. It does model-free Q-Learning updates (direct RL), but also learns a model and uses it for planning (model-based). This combination is called model-based reinforcement learning with direct RL. The Sutton & Barto framework (Chapter 8) shows that all RL methods fall on a spectrum from pure model-free to pure model-based, and Dyna sits in the middle.
What happens if the model is wrong?
In deterministic environments, the model is always correct. But in stochastic environments, the model might misremember or oversimplify. If the model predicts a transition that doesn't match reality, planning will update Q with wrong information. This is why Dyna-Q+ (a variant) adds exploration bonuses to encourage revisiting states and updating the model. In this tool, the maze is deterministic, so the model is always perfect.
Is this really running in my browser?
Yes! Both agents, the maze environment, the model, planning, and visualization all run in pure JavaScript on your device. No server, no API calls, no data upload.