Chapter 6 DPO — Direct Preference Optimization

Motivation

PPO requires 4 models in memory (policy, reference, reward model, value head), complex RL infrastructure, and is notoriously unstable. DPO [302] asks: can we skip the RL and learn directly from preferences?

Key insight: The optimal policy under the RLHF objective (reward maximization + KL penalty) has a closed-form solution. We can derive a supervised loss that implicitly optimizes the same objective.

Mathematical Derivation

Step 1: RLHF objective: $\max_{\pi}\mathbb{E}_{x,y\sim\pi}[r(x,y)]-\beta D_{\text{KL}}[\pi\|\pi_{\text{ref}}]$

Step 2: The optimal solution is: $\pi^{*}(y|x)=\frac{1}{Z(x)}\pi_{\text{ref}}(y|x)\exp\left(\frac{r(x,y)}{\beta}\right)$

Step 3: Rearrange to express reward in terms of policy: $r(x,y)=\beta\log\frac{\pi^{*}(y|x)}{\pi_{\text{ref}}(y|x)}+\beta\log Z(x)$

Step 4: Substitute into Bradley-Terry preference model $P(y_{w}\succ y_{l})=\sigma(r(y_{w})-r(y_{l}))$ . The $Z(x)$ cancels!

\boxed{\mathcal{L}_{\text{DPO}}(\theta)=-\mathbb{E}_{(x,y_{w},y_{l})}\left[\log\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|x)}-\beta\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)}\right)\right]}

(6.1)

Gradient Analysis

The DPO gradient decomposes as:

\nabla_{\theta}\mathcal{L}=-\beta\cdot\underbrace{\sigma(-\hat{r}_{w}+\hat{r}_{l})}_{\text{weight: higher when model is wrong}}\cdot\left[\nabla_{\theta}\log\pi_{\theta}(y_{w}|x)-\nabla_{\theta}\log\pi_{\theta}(y_{l}|x)\right]

(6.2)

Interpretation: The gradient increases probability of chosen and decreases rejected. The weight is largest when the model currently prefers the wrong answer — it focuses learning on “confusing” pairs.

TRL Implementation

The following shows a minimal working example using HuggingFace TRL.

from trl import DPOConfig, DPOTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig
from datasets import load_dataset
 
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct",
    torch_dtype=torch.bfloat16, attn_implementation="flash_attention_2")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
 
# Dataset format: {"prompt": str, "chosen": str, "rejected": str}
dataset = load_dataset("argilla/ultrafeedback-binarized-preferences")
 
lora_config = LoraConfig(r=64, lora_alpha=16, lora_dropout=0.05,
    target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"])
 
dpo_config = DPOConfig(
    output_dir="./dpo_output",
    beta=0.1,                    # KL regularization strength
    learning_rate=5e-7,          # Very low LR for stability
    loss_type="sigmoid",         # Standard DPO loss
    max_length=2048,             # Max sequence length
    max_prompt_length=1024,      # Truncation for prompts
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,  # Effective batch = 16
    gradient_checkpointing=True,
    bf16=True,
    num_train_epochs=1,          # DPO overfits fast - 1 epoch!
    warmup_ratio=0.1,
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=200,
    save_strategy="steps",
    save_steps=500,
)
 
trainer = DPOTrainer(
    model=model,
    ref_model=None,             # With LoRA, ref = base model (no copy needed!)
    args=dpo_config,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    tokenizer=tokenizer,
    peft_config=lora_config,
)
trainer.train()
# Key metrics to monitor: train/rewards/chosen, train/rewards/rejected, train/rewards/margins

How DPO Works: Full Mechanics

This section provides the complete computational details of DPO — what happens at the token level during training.

Sequence-Level Log-Probabilities

The key quantity in DPO is the log-probability of an entire sequence $y=(y_{1},y_{2},\ldots,y_{T})$ given prompt $x$ . This is computed as the sum of per-token log-probabilities:

\boxed{\log\pi_{\theta}(y|x)=\sum_{t=1}^{T}\log\pi_{\theta}(y_{t}\mid x,y_{<t})}

(6.3)

Each term $\log\pi_{\theta}(y_{t}|x,y_{<t})$ is the log-softmax output at position $t$ for the actual token $y_{t}$ in the sequence. This is identical to the cross-entropy loss used in standard language modeling — but here we sum rather than average.

Critical detail: The gradient flows through every token position in both $y_{w}$ and $y_{l}$ . There is no masking of intermediate tokens — every token contributes to the sequence-level log-probability.

The DPO Loss Decomposed

Starting from the loss:

\mathcal{L}_{\text{DPO}}(\theta)=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\!\left[\log\sigma\!\left(\beta\cdot h_{\theta}(x,y_{w},y_{l})\right)\right]

(6.4)

where the “implicit reward margin” $h_{\theta}$ is:

h_{\theta}(x,y_{w},y_{l})=\underbrace{\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|x)}}_{\text{chosen reward proxy}}-\underbrace{\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)}}_{\text{rejected reward proxy}}

(6.5)

Expanding into token-level terms:

\boxed{h_{\theta}=\sum_{t=1}^{|y_{w}|}\!\left[\log\pi_{\theta}(y_{w}^{t}|x,y_{w}^{<t})-\log\pi_{\text{ref}}(y_{w}^{t}|x,y_{w}^{<t})\right]-\sum_{t=1}^{|y_{l}|}\!\left[\log\pi_{\theta}(y_{l}^{t}|x,y_{l}^{<t})-\log\pi_{\text{ref}}(y_{l}^{t}|x,y_{l}^{<t})\right]}

(6.6)

Forward Pass: Step by Step

For one training example $(x,y_{w},y_{l})$ :

Concatenate: Form two sequences: $[x;y_{w}]$ and $[x;y_{l}]$ . Pad to equal length within the batch.
Forward pass (policy $\pi_{\theta}$ ): Run both sequences through the model. Collect logits at every response position.
Extract log-probs: At each position $t$ in the response, take $\log\text{softmax}(\text{logits}_{t})[y_{t}]$ — the log-probability of the actual token.
Sum over tokens:logp_chosen $\displaystyle=\sum_{t\in\text{response positions}}\log\pi_{\theta}(y_{w}^{t}|x,y_{w}^{<t})$ (6.7)logp_rejected $\displaystyle=\sum_{t\in\text{response positions}}\log\pi_{\theta}(y_{l}^{t}|x,y_{l}^{<t})$ (6.8)
Subtract reference (pre-computed or from second forward pass):ratio_w $\displaystyle=\text{logp\_chosen}-\text{ref\_logp\_chosen}$ (6.9)ratio_l $\displaystyle=\text{logp\_rejected}-\text{ref\_logp\_rejected}$ (6.10)
Compute loss: $\mathcal{L}=-\log\sigma(\beta\cdot(\text{ratio\_w}-\text{ratio\_l}))$
Backward pass: Gradients flow back through steps 5 $\rightarrow$ 4 $\rightarrow$ 3 $\rightarrow$ 2 to update $\theta$ .

Token-Level Gradient Analysis

Does every token get a gradient? Yes. The gradient with respect to the logits at position $t$ in the chosen sequence is:

\frac{\partial\mathcal{L}}{\partial\text{logits}_{t}^{(w)}}=-\underbrace{\sigma(-\beta\cdot h_{\theta})}_{\text{scaling factor}}\cdot\beta\cdot\frac{\partial\log\pi_{\theta}(y_{w}^{t}|\cdot)}{\partial\text{logits}_{t}^{(w)}}

(6.11)

Key insight: The scaling factor $\sigma(-\beta\cdot h_{\theta})$ is shared across all tokens in both sequences. It acts as an adaptive learning rate:

When $h_{\theta}$ is small (model can’t distinguish chosen from rejected): scaling $\approx 0.5$ — strong gradient, learn aggressively.
When $h_{\theta}$ is large (model already prefers chosen): scaling $\approx 0$ — negligible gradient, don’t over-fit.

Effect on chosen tokens: Probability is increased (log-prob pushed up).

Effect on rejected tokens: Probability is decreased (log-prob pushed down).

Relative to reference: Only the difference from $\pi_{\text{ref}}$ matters. If the model already assigns high probability to the chosen response (matching the reference), there’s little gradient.

Per-Token vs. Sequence-Level: Length Normalization

A subtle issue: longer sequences naturally have lower log-probabilities (more terms summed, each $\leq 0$ ). If $|y_{w}|\gg|y_{l}|$ , the loss can be biased toward preferring shorter responses.

Solutions:

Length-normalized DPO: Replace $\log\pi_{\theta}(y|x)$ with $\frac{1}{|y|}\sum_{t}\log\pi_{\theta}(y_{t}|\cdot)$ . Used in some implementations (SimPO adopts this).
Standard DPO: Uses raw sum (no normalization). This implicitly penalizes verbosity — the model must assign high probability to every token in the chosen response.
Practical impact: On benchmarks, length-normalized DPO reduces length gaming but can hurt instruction-following quality. Standard (unnormalized) is more common in production.

Label Masking: What Gets Gradients

Pseudocode: DPO Training Step

Common Pitfalls

DPO Variants and When Each Fails

β\beta Selection Guide

$\beta$ RegimeWhen to Use0.01Very aggressiveOnly if data is extremely clean and you need large distributional shift0.05AggressiveGood data, want noticeable improvement over SFT0.1StandardDefault starting point. Good balance of quality vs stability0.2ConservativeNoisy data, or model already close to desired behavior0.5Very conservativeSafety fine-tuning where you must not break capabilities

DPO Batch Size Configuration and Scaling

Unlike standard SFT which operates on single-sequence token predictions, DPO leverages a pairwise loss comparing a preferred sequence against a dispreferred sequence. This fundamentally alters memory utilization and optimization stability.

Global Batch Size Target

Empirical evidence across DPO implementations establishes an optimal global batch size range:

\boxed{B_{\text{global}}\in[32,128]}

(6.12)

$B_{\text{global}}<32$ : Severe gradient noise in implicit reward estimation $\rightarrow$ policy oscillates destructively between alignment goals (helpfulness vs. safety).
$B_{\text{global}}>128$ : Diminishing returns on convergence velocity; high communication overhead across distributed compute.

Mathematical Decomposition

Because DPO loads two model copies simultaneously (active policy $\pi_{\theta}$ + frozen reference $\pi_{\text{ref}}$ ), per-sequence memory is doubled. The global batch size is decomposed as:

\boxed{B_{\text{global}}=B_{\text{micro}}\times N_{\text{GPUs}}\times K_{\text{accum}}}

(6.13)

$B_{\text{micro}}$ : Per-device micro-batch size (preference pairs per forward pass).
$N_{\text{GPUs}}$ : Number of parallel data-processing devices.
$K_{\text{accum}}$ : Gradient accumulation steps before weight update.

The pairing multiplier: A single DPO data instance contains a prompt ( $x$ ), chosen ( $y_{w}$ ), and rejected ( $y_{l}$ ). The actual tensor load per micro-batch:

T_{\text{sequences}}=2\times B_{\text{micro}}

(6.14)

For models $>$ 7B parameters on 80GB GPUs with context lengths 4096–8192 tokens, the physical limit is rigidly constrained to $B_{\text{micro}}\in[1,2]$ .

Distributed Scaling Configurations

Configuration	Single GPU	8-GPU Node
$B_{\text{global}}$	64	64
$B_{\text{micro}}$	2 (4 sequences)	2 (4 sequences)
$N_{\text{GPUs}}$	1	8
$K_{\text{accum}}$	32 steps	4 steps
Throughput	Sequential/slow	High parallel throughput

Table 6.1:Distributed scaling profiles for DPO training ( $B_{\text{global}}=64$ target).

VRAM Optimization: Pre-computing Reference Log-Probabilities

The DPO loss:

\mathcal{L}_{\text{DPO}}(\theta)=-\mathbb{E}_{(x,y_{w},y_{l})}\!\left[\log\sigma\!\left(\beta\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|x)}-\beta\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)}\right)\right]

(6.15)

Because $\pi_{\text{ref}}$ is completely static throughout training, its outputs can be pre-computed:

DPO Extensions and Variants

Direct Preference Optimization (DPO) reformulates RLHF as a supervised learning problem by deriving a closed-form mapping between the reward function and the optimal policy. The standard DPO loss is:

\mathcal{L}_{\text{DPO}}(\theta)=-\mathbb{E}_{(q,y_{w},y_{l})}\!\left[\log\sigma\!\left(\beta\log\frac{\pi_{\theta}(y_{w}|q)}{\pi_{\text{ref}}(y_{w}|q)}-\beta\log\frac{\pi_{\theta}(y_{l}|q)}{\pi_{\text{ref}}(y_{l}|q)}\right)\right],

where $y_{w}$ is the preferred (winning) response, $y_{l}$ is the dispreferred (losing) response, and $\beta$ controls the strength of the KL penalty. The following subsections cover the most important extensions and variants.

f-DPO – Generalised f-Divergence DPO

The f-DPO loss replaces the log-ratio with the derivative of the f-divergence generator:

\mathcal{L}_{f\text{-DPO}}=-\mathbb{E}\!\left[f^{\prime}\!\left(\frac{\pi_{\theta}(y_{w}|q)}{\pi_{\text{ref}}(y_{w}|q)}\right)-f^{\prime}\!\left(\frac{\pi_{\theta}(y_{l}|q)}{\pi_{\text{ref}}(y_{l}|q)}\right)\right],

where $f^{\prime}$ is the derivative of the f-divergence generator function.

Robust DPO

Assume each label is flipped with probability $\epsilon$ (the noise rate). The debiased loss is:

\boxed{\mathcal{L}_{\text{robust}}=\frac{(1-\epsilon)\,\mathcal{L}_{\text{DPO}}(y_{w},y_{l})-\epsilon\,\mathcal{L}_{\text{DPO}}(y_{l},y_{w})}{1-2\epsilon},}

where $\mathcal{L}_{\text{DPO}}(y_{w},y_{l})$ is the standard DPO loss treating $y_{w}$ as preferred, and $\mathcal{L}_{\text{DPO}}(y_{l},y_{w})$ is the loss with labels flipped. This correction removes the bias introduced by label noise.

TR-DPO – Trust Region DPO

TR-DPO updates the reference model using an exponential moving average (EMA):

\pi_{\text{ref}}^{(t+1)}\leftarrow\alpha\cdot\pi_{\theta}^{(t)}+(1-\alpha)\cdot\pi_{\text{ref}}^{(t)},

where $\alpha\in(0,1)$ is the mixup coefficient. This is applied every $T_{\text{sync}}$ gradient steps.

EXO – Exact Optimisation

EXO minimises the reverse KL between the model distribution and the target (reward-optimal) distribution:

\mathcal{L}_{\text{EXO}}=\mathbb{E}_{y\sim\pi_{\theta}}\!\left[\log\frac{\pi_{\theta}(y|q)}{p^{*}(y|q)}\right],

where $p^{*}(y|q)\propto\pi_{\text{ref}}(y|q)\exp(r(y,q)/\beta)$ is the optimal policy. In practice, EXO approximates this using the available preference pairs:

\mathcal{L}_{\text{EXO}}\approx-\mathbb{E}\!\left[\log\sigma\!\left(\beta\log\frac{\pi_{\text{ref}}(y_{w}|q)}{\pi_{\theta}(y_{w}|q)}-\beta\log\frac{\pi_{\text{ref}}(y_{l}|q)}{\pi_{\theta}(y_{l}|q)}\right)\right].

Note the swapped roles of $\pi_{\theta}$ and $\pi_{\text{ref}}$ compared to DPO.

NCA – Noise Contrastive Alignment

NCA reframes alignment as noise-contrastive estimation. The loss has three terms:

\boxed{\mathcal{L}_{\text{NCA}}=-\log\sigma(r_{w})-\tfrac{1}{2}\log\sigma(-r_{w})-\tfrac{1}{2}\log\sigma(-r_{l}),}

where $r_{y}=\beta\log(\pi_{\theta}(y|q)/\pi_{\text{ref}}(y|q))$ is the implicit reward. The first term encourages high reward for $y_{w}$ ; the second and third terms penalise high reward for both $y_{w}$ and $y_{l}$ (preventing collapse).

SLiC-HF – Sequence Likelihood Calibration

The SLiC-HF loss is:

\mathcal{L}_{\text{SLiC}}=\max\!\left(0,\;\delta-\beta\log\frac{\pi_{\theta}(y_{w}|q)}{\pi_{\text{ref}}(y_{w}|q)}+\beta\log\frac{\pi_{\theta}(y_{l}|q)}{\pi_{\text{ref}}(y_{l}|q)}\right),

where $\delta$ is the margin threshold. When the model already assigns a margin of $\delta$ between winning and losing responses, the loss is zero.

Iterative RPO – Reasoning Preference Optimisation

The RPO loss combines DPO and SFT:

\mathcal{L}_{\text{RPO}}=\lambda_{1}\mathcal{L}_{\text{DPO}}(y_{w},y_{l})+\lambda_{2}\mathcal{L}_{\text{NLL}}(y_{w}),

where $\mathcal{L}_{\text{NLL}}(y_{w})=-\log\pi_{\theta}(y_{w}|q)$ is the standard language modelling loss on the winning response.

SimPO – Simple Preference Optimisation

SimPO defines the implicit reward as:

r_{\text{SimPO}}(y|q)=\frac{\beta}{|y|}\log\pi_{\theta}(y|q),

and the loss as:

\boxed{\mathcal{L}_{\text{SimPO}}=-\mathbb{E}\!\left[\log\sigma\!\left(\frac{\beta}{|y_{w}|}\log\pi_{\theta}(y_{w}|q)-\frac{\beta}{|y_{l}|}\log\pi_{\theta}(y_{l}|q)-\gamma\right)\right],}

where $\gamma>0$ is a target reward margin that ensures the winning response has strictly higher reward than the losing response by at least $\gamma$ .