← All problems
#107 PPO Clipped Surrogate Loss Medium

Given precomputed advantages and log probabilities from the current and previous policies, compute the scalar PPO clipped-surrogate loss. For each response token, PPO compares the current policy with the policy that generated the data, limits how far that ratio can move from one, and averages the resulting surrogate objective across the batch.

Implementation Requirements

Loss Calculation

At position [b, s], log_pi and log_pi_old are the log probabilities assigned to the same response token by the current and previous policies. Let A denote advantages, and let epsilon denote clip_eps.

First, compute the probability ratio between the current and previous policies:

\[ r_{b,s} = \exp\!\left(\log\pi_{b,s} - \log\pi^{\mathrm{old}}_{b,s}\right) \]

Clamp that ratio to PPO's trust region:

\[ \widehat{r}_{b,s} = \operatorname{clip}\!\left(r_{b,s}, 1 - \varepsilon, 1 + \varepsilon\right) \]

The per-token surrogate takes the less optimistic of the unclipped and clipped values. This min is intentional: it applies correctly to both positive and negative advantages.

\[ L_{b,s}^{\mathrm{CLIP}} = \min\!\left(r_{b,s} A_{b,s},\; \widehat{r}_{b,s} A_{b,s}\right) \]

PPO maximizes this surrogate objective. Because this challenge returns a loss for minimization, negate its mean over every batch and sequence position:

\[ \text{output}[0] = -\frac{1}{B S}\sum_{b=0}^{B-1}\sum_{s=0}^{S-1} L_{b,s}^{\mathrm{CLIP}} \]

Examples

For B = 1, S = 4, and clip_eps = 0.2:

\[ \begin{aligned} \text{advantages} &= \begin{bmatrix} 1.0 & -2.0 & 3.0 & -4.0 \end{bmatrix} \\[6pt] \text{log_pi} &= \begin{bmatrix} \log(1.3) & \log(0.7) & \log(1.1) & \log(0.8) \end{bmatrix} \\[6pt] \text{log_pi_old} &= \begin{bmatrix} 0.0 & 0.0 & 0.0 & 0.0 \end{bmatrix} \end{aligned} \]

The intermediate values are:

\[ \begin{aligned} r &= \begin{bmatrix} 1.3 & 0.7 & 1.1 & 0.8 \end{bmatrix} \\[6pt] \widehat{r} &= \begin{bmatrix} 1.2 & 0.8 & 1.1 & 0.8 \end{bmatrix} \\[6pt] L^{\mathrm{CLIP}} &= \begin{bmatrix} 1.2 & -1.6 & 3.3 & -3.2 \end{bmatrix} \end{aligned} \] \[ \text{output}[0] = -\operatorname{mean}\!\left(L^{\mathrm{CLIP}}\right) = 0.075 \]

Constraints

Reference

Proximal Policy Optimization Algorithms, Schulman et al. (2017)

Guided walkthrough coming soon. Below is the official starter signature in each language; the clickable line-by-line explanation hasn't been authored yet.
Problem text & starters from the official AlphaGPU/leetgpu-challenges. Solve it on leetgpu.com.