Given precomputed advantages and log probabilities from the current and previous policies, compute the scalar PPO clipped-surrogate loss. For each response token, PPO compares the current policy with the policy that generated the data, limits how far that ratio can move from one, and averages the resulting surrogate objective across the batch.
solve(advantages, log_pi, log_pi_old, output, clip_eps, B, S) with the signature unchangedoutput[0][B, S] and the output has shape [1]S positions are valid response tokens; padding and token masks are outside the scope of this challengeadvantages and log_pi_old are fixed precomputed inputs; do not differentiate through them
At position [b, s], log_pi and log_pi_old are the log
probabilities assigned to the same response token by the current and previous policies. Let
A denote advantages, and let epsilon denote
clip_eps.
First, compute the probability ratio between the current and previous policies:
\[ r_{b,s} = \exp\!\left(\log\pi_{b,s} - \log\pi^{\mathrm{old}}_{b,s}\right) \]Clamp that ratio to PPO's trust region:
\[ \widehat{r}_{b,s} = \operatorname{clip}\!\left(r_{b,s}, 1 - \varepsilon, 1 + \varepsilon\right) \]
The per-token surrogate takes the less optimistic of the unclipped and clipped values. This
min is intentional: it applies correctly to both positive and negative advantages.
PPO maximizes this surrogate objective. Because this challenge returns a loss for minimization, negate its mean over every batch and sequence position:
\[ \text{output}[0] = -\frac{1}{B S}\sum_{b=0}^{B-1}\sum_{s=0}^{S-1} L_{b,s}^{\mathrm{CLIP}} \]For B = 1, S = 4, and clip_eps = 0.2:
The intermediate values are:
\[ \begin{aligned} r &= \begin{bmatrix} 1.3 & 0.7 & 1.1 & 0.8 \end{bmatrix} \\[6pt] \widehat{r} &= \begin{bmatrix} 1.2 & 0.8 & 1.1 & 0.8 \end{bmatrix} \\[6pt] L^{\mathrm{CLIP}} &= \begin{bmatrix} 1.2 & -1.6 & 3.3 & -3.2 \end{bmatrix} \end{aligned} \] \[ \text{output}[0] = -\operatorname{mean}\!\left(L^{\mathrm{CLIP}}\right) = 0.075 \]B ≤ 256S ≤ 16,3840 ≤ clip_eps < 1log_pi - log_pi_old lies in [-16, 16], so every exponential is finite in float32B = 256, S = 16,384Proximal Policy Optimization Algorithms, Schulman et al. (2017)