SAC — Soft Actor-Critic for Continuous Control
Everything so far used discrete actions (left/right). Real robots need continuous control — motor torque, steering angle, voltage. SAC handles this with three innovations: (1) Maximum entropy — the agent optimizes reward PLUS policy entropy, balancing exploitation and exploration automatically. (2) Stochastic policy — the Actor outputs a Gaussian distribution (mean + std), not a single action. (3) Twin Q-networks — take the min of two critics to fight overestimation. Watch a pendulum swing up from scratch, purely in your browser. TensorFlow.js, 100% browser-side.
Policy Distribution π(a|s)
⚙️ Hyperparameters
Episode Reward (higher = better, max ≈ 0)
Policy Entropy (exploration over time)
Starts high (exploring), should decrease (exploiting). If it drops too fast, the agent may get stuck.
SAC: Continuous Control with Maximum Entropy
1. Why Continuous Actions?
DQN's max_a Q(s,a) requires enumerating all actions. With 2 actions (CartPole), that's trivial. With continuous torque ∈ [-2, 2], there are infinite actions — you can't take the max. SAC solves this by learning a policy π(a|s) that directly outputs continuous actions, bypassing the max entirely.
2. Maximum Entropy
SAC doesn't just maximize reward — it maximizes reward + α·entropy. The objective: J = E[Σ γⁿ(rₙ + α·H(π))]. Entropy measures how "random" the policy is. Higher entropy = more exploration. The weight α controls the explore-exploit tradeoff automatically — no ε-greedy schedule to tune.
3. Stochastic Policy
The Actor doesn't output a single action. It outputs a Gaussian distribution (mean μ + std σ). During training, actions are sampled: a = tanh(μ + σ·ε), where ε ~ N(0,1). This stochasticity is the exploration mechanism — no separate ε-greedy needed. The reparameterization trick makes this differentiable.
4. Twin Q-Networks
Two independent Q-networks (Q₁, Q₂) are trained on the same data. SAC uses min(Q₁, Q₂) everywhere — in the target, in the actor loss, everywhere. This combats the overestimation bias that plagues single-Q methods (same idea as Double DQN, but simpler). The min is pessimistic, which is good — it prevents the agent from being overly optimistic about bad actions.
5. Off-Policy + Replay Buffer
SAC is off-policy: it can learn from data collected by any policy (old versions of itself). A replay buffer stores (s, a, r, s') transitions. Each training step samples a random mini-batch. This means every transition can be used multiple times — dramatically better data efficiency than PPO/A2C.
6. Reparameterization Trick
Sampling a ~ π(·|s) is not differentiable (you can't backprop through randomness). The fix: a = μ + σ·ε where ε is random noise. Now the gradient flows through μ and σ, not through ε. This is the reparameterization trick — the same technique used in VAEs. The tanh squashes the action to [-1, 1], and a log-prob correction accounts for the nonlinear transformation.
Frequently Asked Questions
What is the maximum entropy framework?
In standard RL, the objective is to maximize expected return: J = E[Σ γⁿ rₙ]. SAC adds an entropy bonus: J = E[Σ γⁿ (rₙ + α·H(π(·|sₙ)))]. Entropy H measures the "randomness" of the policy — a uniform distribution has high entropy, a deterministic policy has zero. By rewarding entropy, SAC encourages the agent to keep exploring and not commit too early. The weight α controls the tradeoff: α=0 is pure exploitation, α→∞ is pure exploration. The "soft" in SAC refers to this softened objective.
Why does SAC use tanh squashing?
The Gaussian distribution has infinite support — it can sample any real number. But action spaces are bounded (e.g., torque ∈ [-2, 2]). SAC applies tanh to squash the Gaussian sample to [-1, 1], then scales to the action range. The tanh changes the probability density, so the log-probability must be corrected: log π(a|s) = log N(z; μ, σ) - Σ log(1 - tanh²(z)), where z is the pre-tanh sample. This correction ensures the policy remains a valid probability distribution.
What is the reparameterization trick?
To train the Actor, we need gradients of Q(s, a) - α·log π(a|s) with respect to the Actor's parameters. But a is sampled from π, and sampling is not differentiable. The reparameterization trick rewrites the sample as a = μ + σ·ε where ε ~ N(0,1) is treated as a constant (no gradient through it). Now the gradient flows through μ and σ, which ARE differentiable. This is the same trick used in Variational Autoencoders (VAEs).
Why twin Q-networks instead of one?
Single Q-networks suffer from overestimation bias: the max operator tends to amplify noise upward. Even without explicit maximization (SAC uses sampled actions), the Q-values can be overly optimistic. Using two independently-initialized Q-networks and taking the min(Q₁, Q₂) provides a pessimistic bound that counteracts overestimation. This is the same principle as Double DQN/Double Q-Learning, but applied more broadly — in the target, in the actor loss, everywhere.
What is the soft target update?
Instead of copying Q-network weights to target weights every N steps (hard update, like DQN), SAC uses Polyak averaging: θ_target ← τ·θ + (1-τ)·θ_target, where τ is small (0.005). This slowly blends the online network into the target, providing a smooth and stable target. Every training step, the target moves 0.5% toward the online network. This is more stable than hard updates and eliminates the need to tune the update frequency.
How is SAC different from DDPG?
DDPG (Deep Deterministic Policy Gradient) also handles continuous actions, but its Actor is deterministic — it outputs a single action, not a distribution. This means DDPG needs external exploration noise (Ornstein-Uhlenbeck process) which is hard to tune. SAC's stochastic policy provides built-in exploration. Additionally, DDPG uses a single Q-network (no twin), making it vulnerable to overestimation. SAC is generally more stable and sample-efficient than DDPG.
What is automatic entropy tuning?
Instead of fixing α, SAC can learn it. The idea: maintain a target entropy (often -dim(action_space)), and update α to achieve it. If the policy entropy is above target, α decreases (less exploration bonus needed). If below target, α increases (encourage more exploration). This auto-tuning removes a hyperparameter and adapts during training. Our tool uses fixed α for simplicity, but the blog post explains auto-tuning in detail.
Why is Pendulum harder than CartPole?
CartPole is a balancing task — start near upright, stay upright. Pendulum is a swing-up task — start from a random angle (possibly hanging down), and you must swing to gain energy, then balance at the top. The swing-up requires a sequence of torques that build momentum — a single "good" action isn't enough. The reward is always negative (cost-based), and the best possible is around -16 per episode. This makes credit assignment much harder than CartPole's binary reward.