SARSA vs Q-Learning — Cliff Walking
Watch SARSA and Q-Learning learn side-by-side on the classic Cliff Walking benchmark! See on-policy vs off-policy RL diverge in real-time — SARSA learns safe paths, Q-Learning learns optimal but risky paths. Adjust hyperparameters, compare learning curves, and understand the core difference in RL. 100% browser-side.
⚙️ Hyperparameters (shared)
Learning Curves — Reward per Episode
Higher (closer to 0) is better. SARSA learns safer paths; Q-Learning learns optimal but risky paths.
SARSA vs Q-Learning: On-Policy vs Off-Policy
1. The Cliff
A 4×12 grid. Start (S) at bottom-left, Goal (★) at bottom-right. The bottom row between them is a cliff — stepping off costs -100 and sends the agent back to start.
2. SARSA (on-policy)
Updates Q using the actual next action chosen by the ε-greedy policy. It learns the value of the policy it actually follows — including random exploration.
3. Q-Learning (off-policy)
Updates Q using the maximum next Q-value, assuming the agent will act optimally. It learns the optimal policy, even though it explores randomly during training.
4. The Key Difference
SARSA: Q(s,a) ← Q(s,a) + α[r + γ·Q(s',a') − Q(s,a)]Q-Learning: Q(s,a) ← Q(s,a) + α[r + γ·max Q(s',a') − Q(s,a)]
5. Why SARSA is Safer
SARSA knows it might randomly step off the cliff (ε% chance), so it learns to stay away from the edge. Q-Learning assumes it'll never stumble — so it walks right along the cliff edge.
6. Watch Them Diverge
As training progresses, Q-Learning's policy arrows hug the cliff edge (optimal but risky). SARSA's arrows move up, away from the cliff (suboptimal but safe). The learning curves tell the story.
Frequently Asked Questions
What is the difference between SARSA and Q-Learning?
SARSA is on-policy: it learns the Q-values for the policy it actually follows (including ε-greedy exploration). Q-Learning is off-policy: it learns the Q-values for the optimal (greedy) policy, while following an exploratory policy. The mathematical difference is that SARSA uses Q(s',a') (the actual next action) while Q-Learning uses max Q(s',a') (the best possible next action).
Why does Q-Learning walk along the cliff edge?
Q-Learning's update uses max Q(s',a'), which assumes the agent will act optimally in the future. So even though the agent is next to the cliff, Q-Learning assumes it will never accidentally step off — and the shortest path is right along the edge. During training, the ε-greedy behavior causes occasional falls, but the learned Q-values still reflect the optimal path.
Why does SARSA take the longer path?
SARSA's update uses Q(s',a') where a' is the actual next action chosen by ε-greedy. This means SARSA "knows" that with probability ε, it will take a random action — which might be "down" (into the cliff). So SARSA's Q-values account for the risk of random exploration, and the agent learns to move away from the cliff edge to avoid accidental falls.
Which algorithm is better?
Neither is universally "better." Q-Learning learns the optimal policy, which is ideal if you can turn off exploration at deployment. SARSA learns a safer policy that accounts for ongoing exploration, which is better when the agent must keep exploring (e.g., non-stationary environments). In the Cliff Walking, Q-Learning's optimal path is shorter but riskier; SARSA's path is longer but safer.
What does "on-policy" and "off-policy" mean?
On-policy algorithms (like SARSA) evaluate and improve the same policy they use to generate actions. Off-policy algorithms (like Q-Learning) learn about a target policy (e.g., the optimal greedy policy) while following a different behavior policy (e.g., ε-greedy). Off-policy methods can learn from data generated by any policy, including demonstrations — this is crucial for real-world RL.
What is the Cliff Walking environment?
It's a classic RL benchmark from Sutton & Barto's Reinforcement Learning: An Introduction (Example 6.6). A 4×12 grid where the agent must travel from the bottom-left to the bottom-right, with a cliff (-100 penalty) in between. The optimal path goes right along the cliff edge; the safe path goes up and around. It's the textbook example for comparing on-policy vs off-policy learning.
Is this really running in my browser?
Yes! Both SARSA and Q-Learning run in pure JavaScript on your device. No server, no API calls, no data upload. You can disconnect your internet after the page loads.
Why are the Q-values negative?
Every step costs -1, and falling off the cliff costs -100. There's no positive reward — the agent's goal is to minimize total negative reward by reaching the goal as quickly as possible. The Q-values represent expected future costs, so they're naturally negative. The "best" action in each state is the one with the least negative Q-value.