PPO — Proximal Policy Optimization (OpenAI's Default)
A2C throws away data after one update. PPO reuses it! By clipping the policy ratio r = π_new/π_old to [1-ε, 1+ε], PPO safely performs multiple epochs on the same batch — getting more learning per episode without destabilizing. This is OpenAI's go-to algorithm (and the RLHF engine behind ChatGPT). Watch A2C vs PPO train side by side: PPO learns faster, more stably, and recovers from bad episodes. TensorFlow.js, 100% browser-side.
⚙️ Hyperparameters (shared by both agents)
Episode Reward (higher = better)
Policy Ratio Distribution (PPO only)
r = π_new(a|s) / π_old(a|s). PPO clips r to [1−ε, 1+ε] (red zone). Bars outside the green zone are clipped.
PPO: Don't Throw Away Your Data
1. The Data Efficiency Problem
A2C and REINFORCE are on-policy: data collected by the old policy becomes invalid after one update. You play an episode, learn once, throw it away. This is wasteful — each episode cost real computation to generate. Imagine reading a textbook once and never reviewing it.
2. The Naive Fix: Importance Sampling
Mathematically, you can reuse old data with importance sampling: weight each gradient by the ratio r = π_new(a|s) / π_old(a|s). If the new policy assigns 2× the probability to an action compared to the old policy, you weight that sample 2×. But if the policies diverge too much, r explodes to 100× or 0.01× — the gradient becomes enormous and training collapses.
3. PPO's Solution: Clipping
PPO simply clips the ratio to [1−ε, 1+ε] (typically ε=0.2). If the new policy moves more than 20% away from the old, the gradient is cut off. The objective becomes: min(r·A, clip(r, 1−ε, 1+ε)·A). This is the clipped surrogate objective — simple, effective, and surprisingly hard to beat.
4. Multiple Epochs Per Batch
With clipping in place, PPO can safely perform K epochs (typically 4-10) on the same batch of data. A2C does 1 update per episode; PPO does 4. That's 4× more learning per data point — for free. The clip ensures that later epochs don't over-optimize: once the policy has moved 20%, further updates in that direction are suppressed.
5. The Math: Clipped Surrogate
The standard policy gradient objective is E[log π_new(a|s) · A]. PPO rewrites this using the ratio: E[r · A]. Then clips: L^CLIP = E[min(r·A, clip(r, 1−ε, 1+ε)·A)]. When A > 0 (good action), the objective increases r up to 1+ε then stops. When A < 0 (bad action), the objective decreases r down to 1−ε then stops. The policy can only move 20% per epoch — a trust region.
6. Why PPO Won the RL Wars
PPO strikes the perfect balance between simplicity and performance. It's easier to implement than TRPO (which solves a constrained optimization problem), more stable than A2C (which can collapse from one bad update), and more data-efficient than REINFORCE. OpenAI uses it as their default RL algorithm, and it's the core of RLHF — the technique that turned GPT-3 into ChatGPT. If you learn one deep RL algorithm, make it PPO.
Frequently Asked Questions
What is the policy ratio r(θ)?
The policy ratio measures how much the new policy (π_new) differs from the old policy (π_old) for a specific state-action pair: r = π_new(a|s) / π_old(a|s). If r = 1, the policies are identical. If r = 2, the new policy is twice as likely to choose action a. If r = 0.5, it's half as likely. PPO uses this ratio to reweight old data — instead of collecting fresh data for every update, you adjust the gradient by how much the policy has changed.
Why clip at 0.2? What does ε control?
ε (epsilon) controls the maximum allowed policy change per epoch. The standard value is 0.2, meaning the policy can move at most 20% closer to or away from any action per update. Too small (ε=0.05): training is very stable but slow. Too large (ε=0.5): training is fast but can collapse — the policy moves too far and the advantage estimates (computed under the old policy) become invalid. 0.2 is a sweet spot found empirically by OpenAI. In our tool, you can adjust ε and watch how it affects stability.
What is the clipped surrogate objective?
The objective function PPO maximizes: L^CLIP = E[min(r·A, clip(r, 1−ε, 1+ε)·A)]. The min is crucial — it takes the pessimistic bound. If the advantage A is positive (good action), clipping limits how much you benefit from increasing r. If A is negative (bad action), clipping limits how much you benefit from decreasing r. This asymmetry prevents the optimizer from exploiting estimation errors in A — it can only profit from "safe" improvements.
How is PPO different from TRPO?
TRPO (Trust Region Policy Optimization) constrains the KL divergence between old and new policies to be below a threshold δ. It requires solving a constrained optimization problem with conjugate gradient — computationally expensive and complex to implement. PPO achieves a similar effect with a simple clip — no second-order optimization, no KL computation, just min(r·A, clip(r)·A). PPO is simpler, faster, and empirically performs just as well. TRPO is now mostly of historical interest.
How many epochs should I use?
Typically 4-10. Too few (1-2): PPO degenerates to A2C — no data reuse benefit. Too many (20+): the policy drifts too far from the old policy, and even with clipping, the advantage estimates become stale. The standard is 4 for simple environments like CartPole, up to 10 for more complex ones. In our tool, you can adjust K and see how it affects training speed and stability.
What is GAE (Generalized Advantage Estimation)?
GAE is a technique for computing advantages that balances bias and variance. It uses an exponential weighted average of n-step TD advantages: A^GAE_γ,λ = Σ(γλ)^l · δ_{t+l}, where δ = r + γV(s') − V(s). λ=0 gives pure TD (1-step, high bias, low variance); λ=1 gives Monte Carlo (no bias, high variance). λ=0.95 is a common default. Our tool uses Monte Carlo advantage (A = G − V(s)) for simplicity, but the blog post explains GAE in detail.
How does PPO relate to ChatGPT?
ChatGPT uses RLHF (Reinforcement Learning from Human Feedback) to align the model with human preferences. The RL step of RLHF uses PPO: the language model is the "policy", prompts are "states", generated text is "actions", and a reward model (trained on human ratings) provides the "reward". PPO fine-tunes the language model to generate higher-reward text while staying close to the original model (the clip prevents the model from drifting too far into gibberish that happens to score high). So yes — PPO is literally the algorithm behind ChatGPT.
Is PPO on-policy or off-policy?
PPO is technically on-policy — the data must be collected by a policy that's "close enough" to the current one. The clip enforces this closeness. However, PPO is "semi-on-policy": it can reuse data for K epochs, which is a middle ground between pure on-policy (1 epoch) and off-policy (unlimited reuse). If you collect data with policy π_1 and try to use it after policy has drifted to π_10 (10 epochs later), the clip would suppress most updates — the data is effectively useless. So PPO is on-policy with a short memory.