← All problems
#108 DPO Sequence Loss Medium

Given precomputed sequence log probabilities for chosen and rejected responses under a current policy and a reference policy, compute the Direct Preference Optimization (DPO) loss. Each input is a one-dimensional tensor of shape [B]. The output is one value: the DPO loss averaged across all B response pairs.

Each batch entry represents one prompt and its chosen/rejected response pair. Token-level log probabilities have already been summed, and the same tokenization and masking convention was used for the current and reference policies.

Before training, a reviewer labels one response as preferred and one as rejected. Each DPO update then consumes that fixed example: prompt x, chosen response y+, and rejected response y-. The current and reference policies each score both responses, producing four sequence log probabilities.

DPO Offline, before training: human feedback labels the preferred y+ and rejected y- response fixed preference example prompt x, chosen y+, rejected y- current policy trainable weights θ reference policy frozen weights θ_ref l+ = log piθ(y+ | x) l- = log piθ(y- | x) l+ref = log pi_ref(y+ | x) l-ref = log pi_ref(y- | x) z = beta[(l+ - l-) - (l+ref - l-ref)] output[0] = mean(softplus(-z)) across B preference examples only the current-policy weights are updated; the reference policy stays frozen

Implementation Requirements

Loss Calculation

At batch index i, let l+ and l- be the chosen and rejected log probabilities under the current policy. Let l+_ref and l-_ref be the corresponding values under the reference policy.

\[ \Delta_i^{\pi} = \ell_i^{+} - \ell_i^{-} \] \[ \Delta_i^{\mathrm{ref}} = \ell_i^{+,\mathrm{ref}} - \ell_i^{-,\mathrm{ref}} \]

Subtract the reference margin from the current-policy margin, then scale by beta:

\[ z_i = \beta\left(\Delta_i^{\pi} - \Delta_i^{\mathrm{ref}}\right) \]

DPO maximizes the log-sigmoid of this preference logit. This challenge returns the equivalent loss for minimization, averaged across the batch:

\[ \text{output}[0] = -\frac{1}{B}\sum_{i=0}^{B-1} \log\!\left(\sigma(z_i)\right) = \frac{1}{B}\sum_{i=0}^{B-1} \operatorname{softplus}(-z_i) \]

The softplus form is mathematically equivalent and remains finite for very large logits.

Examples

For B = 4 and beta = 0.1:

\[ \begin{aligned} \text{chosen_logps} &= \begin{bmatrix} 0.0 & 1.0 & -1.0 & 2.0 \end{bmatrix} \\[6pt] \text{rejected_logps} &= \begin{bmatrix} 0.0 & 0.0 & 0.0 & 0.0 \end{bmatrix} \\[6pt] \text{chosen_ref_logps} &= \begin{bmatrix} 0.0 & 0.0 & 0.0 & 0.0 \end{bmatrix} \\[6pt] \text{rejected_ref_logps} &= \begin{bmatrix} 0.0 & 0.0 & 0.0 & 0.0 \end{bmatrix} \end{aligned} \]

The reference margin is zero, so the preference logits and final loss are:

\[ z = \begin{bmatrix} 0.0 & 0.1 & -0.1 & 0.2 \end{bmatrix} \] \[ \text{output}[0] = \operatorname{mean}\!\left(\operatorname{softplus}(-z)\right) = 0.67002 \]

Constraints

Reference

Direct Preference Optimization: Your Language Model is Secretly a Reward Model, Rafailov et al. (2023)

Guided walkthrough coming soon. Below is the official starter signature in each language; the clickable line-by-line explanation hasn't been authored yet.
Problem text & starters from the official AlphaGPU/leetgpu-challenges. Solve it on leetgpu.com.