For each of B prompts, GRPO compares the rewards of G sampled responses.
It uses each response's reward relative to the rest of its group as an advantage, rather than
learning a separate critic. That advantage is applied to every one of the response's
S tokens in a clipped policy objective with a reference-policy KL regularizer.
At position [b, g, s], log_pi, log_pi_old, and
log_ref all score the same sampled token y[b,g,s], conditioned on
prompt b and the preceding response tokens.
solve(rewards, log_pi, log_pi_old, log_ref, output, clip_eps, beta, B, G, S) with the signature unchangedoutput[0][B, G]; all log-probability tensors have shape [B, G, S]S positions participate in the reductionlog_pi_old, and log_ref are fixed precomputed inputs; do not differentiate through them
For one prompt b, normalize the G response rewards using the group's
population mean and standard deviation. Let R denote rewards.
This is an outcome-level GRPO advantage. The same advantage is broadcast to every token of
response g:
Compare the current policy with the old policy that sampled the response.
\[ r_{b,g,s} = \exp\!\left(\log\pi_{b,g,s} - \log\pi^{\mathrm{old}}_{b,g,s}\right) \]
Clamp that ratio around one, then take the less optimistic of the clipped and unclipped values.
Let epsilon denote clip_eps. This min applies correctly to
both positive and negative advantages.
GRPO also penalizes divergence from the frozen reference policy. Let beta be the KL
coefficient. The non-negative k3 sample estimator is:
GRPO maximizes the clipped objective minus this penalty. Because this challenge returns a loss for
minimization, write the negated mean over all prompts, responses, and tokens to output[0]:
For B = 1, G = 2, S = 2, clip_eps = 0.2,
and beta = 0.01:
B ≤ 256G ≤ 32S ≤ 16,3840 ≤ clip_eps < 1 and 0 ≤ beta ≤ 1log_pi - log_pi_old and log_ref - log_pi lie in [-16, 16], so every exponential is finite in float32B = 64, G = 16, S = 4,096