Given precomputed sequence log probabilities for chosen and rejected responses under a current
policy and a reference policy, compute the Direct Preference Optimization (DPO) loss. Each input
is a one-dimensional tensor of shape [B]. The output is one value: the DPO loss
averaged across all B response pairs.
Each batch entry represents one prompt and its chosen/rejected response pair. Token-level log probabilities have already been summed, and the same tokenization and masking convention was used for the current and reference policies.
Before training, a reviewer labels one response as preferred and one as rejected. Each DPO update
then consumes that fixed example: prompt x, chosen response y+, and
rejected response y-. The current and reference policies each score both responses,
producing four sequence log probabilities.
solve(chosen_logps, rejected_logps, chosen_ref_logps, rejected_ref_logps, output, beta, B) with the signature unchangedoutput[0][B]softplus(-logits) or an equivalent formulation
At batch index i, let l+ and l- be the chosen and rejected
log probabilities under the current policy. Let l+_ref and l-_ref be the
corresponding values under the reference policy.
Subtract the reference margin from the current-policy margin, then scale by beta:
DPO maximizes the log-sigmoid of this preference logit. This challenge returns the equivalent loss for minimization, averaged across the batch:
\[ \text{output}[0] = -\frac{1}{B}\sum_{i=0}^{B-1} \log\!\left(\sigma(z_i)\right) = \frac{1}{B}\sum_{i=0}^{B-1} \operatorname{softplus}(-z_i) \]The softplus form is mathematically equivalent and remains finite for very large logits.
For B = 4 and beta = 0.1:
The reference margin is zero, so the preference logits and final loss are:
\[ z = \begin{bmatrix} 0.0 & 0.1 & -0.1 & 0.2 \end{bmatrix} \] \[ \text{output}[0] = \operatorname{mean}\!\left(\operatorname{softplus}(-z)\right) = 0.67002 \]B ≤ 65,5360 < beta ≤ 1B = 65,536