Question 9: The Direct Preference Optimization (DPO) loss function is…

The Direct Preference Optimization (DPO) loss function is given as:

$L_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y_w, y_l) \sim D}\left[\log \sigma\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)\right]$

where,

• $\pi_\theta(y \mid x)$: Probability assigned to an output $y$ by the trained policy (fine-tuned model) for a given input $x$.
• $\pi_{\text{ref}}(y \mid x)$: Probability assigned by the reference policy (pretrained model).
• $(x, y_w, y_l)$: Input $x$ with a preferred output $y_w$ and a less preferred output $y_l$.
• $\beta$: A temperature parameter controlling the sensitivity to log-probability ratios.
• $\sigma(z) = \frac{1}{1+e^{-z}}$: The logistic sigmoid function.

Which of the following statements about the terms in the DPO loss function are correct?