Monte Carlo Blackjack โ Learn the Optimal Strategy
Watch Monte Carlo control learn the optimal Blackjack strategy from scratch! The agent plays thousands of hands per second, using first-visit MC to estimate Q-values. See the strategy table converge to the famous optimal Blackjack playbook. Compare with the known optimal policy. 100% browser-side, no server needed.
No Usable Ace
Usable Ace
Win Rate Over Episodes
โ๏ธ Hyperparameters
Monte Carlo Control: Learning Blackjack by Playing
1. The MC Idea
Monte Carlo methods learn from complete episodes. Instead of estimating future rewards (like TD learning), MC waits for the episode to finish, then uses the actual return to update Q-values. No bootstrapping โ no guessing from estimates.
2. Why Blackjack?
Blackjack is the textbook MC environment (Sutton & Barto Example 5.1). The reward only comes at the end of each hand โ there's no intermediate reward. This makes it perfect for MC: you must wait for the hand to finish to know if you won or lost.
3. First-Visit MC
For each state-action pair in an episode, we only update on the first visit (not repeated visits). The Q-value becomes the average of all returns observed from that state-action. This is an unbiased estimator of the true expected return.
4. ฮต-Greedy Control
The agent uses ฮต-greedy: with probability ฮต it explores (random action), otherwise it exploits (best known action). ฮต decays over time, shifting from exploration to exploitation. The policy improves as Q-values become more accurate.
5. The Strategy Table
The 10ร10 grid shows the learned policy: rows = player sum (12-21), columns = dealer's up card (Ace-10). Green = Hit, Red = Stand. Watch it converge to the famous optimal Blackjack strategy as episodes accumulate.
6. MC vs TD
MC has zero bias (uses real returns) but high variance (each episode is different). TD has bias (uses estimates) but low variance (updates every step). MC only works for episodic tasks; TD works for continuing tasks too. Blackjack is inherently episodic โ perfect for MC.
Frequently Asked Questions
What is the difference between Monte Carlo and TD learning?
TD learning (like Q-Learning) updates Q-values after every step, using a mix of the immediate reward and an estimated future value (bootstrapping). Monte Carlo waits for the entire episode to finish, then uses the actual total return. MC has no bias (no estimates involved) but higher variance (one full episode per update). TD has bias but lower variance (updates every step). In blackjack, the reward only comes at the end, so MC is a natural fit.
Why is ฮณ (gamma) not used here?
In blackjack, the reward only arrives at the very end of the episode (win +1, draw 0, lose -1). There are no intermediate rewards, and the episode is short (a few steps). So we use ฮณ = 1 (no discounting) โ the return G is simply the final reward. Discounting matters more for long or continuing tasks where future rewards should be worth less than immediate ones.
What does "usable ace" mean?
An ace can count as 1 or 11. A usable ace is one that counts as 11 without busting (e.g., Ace + 6 = 17, not 7). If counting the ace as 11 would bust, it converts to 1. The two strategy tables are separate because having a usable ace changes the optimal strategy โ you can hit more aggressively since the ace can always drop to 1.
What is "first-visit" vs "every-visit" MC?
If a state-action pair appears multiple times in one episode, first-visit MC only counts the first occurrence's return. Every-visit MC counts all occurrences. Both converge to the true Q-value, but first-visit is the standard textbook approach. We use first-visit here.
Why doesn't the win rate reach 50%?
Blackjack has a built-in house edge. Even with perfect play, the dealer wins more often than the player (roughly 42% win, 9% draw, 49% lose). The optimal strategy minimizes losses โ it doesn't guarantee winning. A ~42% win rate with ~9% draws is about as good as it gets.
How does this compare to the optimal strategy?
Enable "Compare Optimal" to see the known optimal Blackjack strategy (from Sutton & Barto). As training progresses, the learned policy should converge to this strategy. Minor differences are normal due to the stochastic nature of MC and limited samples in some states (like player sum 20-21, where the agent rarely visits).
What is the dealer's strategy?
The dealer follows a fixed policy: hit until the hand totals 17 or more, then stand. This is the standard casino rule (stand on all 17s). The dealer has no choices โ the only decision-maker is the player.
Is this really running in my browser?
Yes! The entire blackjack environment, MC control algorithm, and visualization run in pure JavaScript on your device. No server, no API calls, no data upload. The agent plays thousands of hands per second using your CPU.