Chapter 8 Preference Optimization Variants

This chapter covers the family of methods that extend or replace DPO with different objectives, data assumptions, or architectural trade-offs. Each addresses a specific limitation of standard offline DPO: distribution shift (Online DPO), the need for paired data (KTO), overfitting to noisy labels (IPO), reference model memory cost (ORPO), or training complexity (Best-of-N).

Online DPO

Motivation

Standard DPO’s primary limitation: the preference data was generated by a different model (often an older checkpoint or even a different model family). As training progresses, the policy generates text that looks nothing like the training pairs $\rightarrow$ the loss is optimizing on an irrelevant distribution.

Online DPO solution[127]: Generate fresh preference pairs from the current policy at every step, judge them with a reward model, then apply the DPO loss.

Algorithm

Generate $K$ responses per prompt from current $\pi_{\theta}$
Score all responses with reward model $r_{\phi}$
Create pairs: highest-scoring = chosen, lowest-scoring = rejected
Apply DPO loss on these fresh pairs
Repeat (new generation every step)

TRL Implementation

The following shows a minimal working example using HuggingFace TRL.

from trl import OnlineDPOConfig, OnlineDPOTrainer
from transformers import AutoModelForCausalLM, AutoModelForSequenceClassification
 
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct",
    torch_dtype=torch.bfloat16)
reward_model = AutoModelForSequenceClassification.from_pretrained(
    "RLHFlow/ArmoRM-Llama3-8B-v0.1", torch_dtype=torch.bfloat16)
 
online_dpo_config = OnlineDPOConfig(
    output_dir="./online_dpo_output",
    learning_rate=5e-7,
    beta=0.1,                    # DPO beta (same meaning as standard DPO)
    num_generations=4,           # K responses per prompt
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    max_new_tokens=512,
    temperature=0.7,
    bf16=True,
    num_train_epochs=1,
    logging_steps=10,
)
 
trainer = OnlineDPOTrainer(
    model=model,
    reward_model=reward_model,
    args=online_dpo_config,
    train_dataset=prompt_dataset,
    tokenizer=tokenizer,
)
trainer.train()

Online DPO vs Offline DPO vs PPO

DataModelsLossBest ForOffline DPOStatic pairs2 (policy + reference)DPOQuick alignment, limited computeOnline DPOFresh from $\pi_{\theta}$ 3 (policy + reference + reward model)DPOWhen DPO plateaus, need explorationPPOFresh from $\pi_{\theta}$ 4 (policy + reference + reward model + value head)PPO clipMax quality, complex reasoning

KTO — Kahneman-Tversky Optimization

Motivation

DPO requires paired preferences: for the same prompt, you need both a good and bad response. In practice, most feedback is unpaired: users give thumbs up/down on individual responses, with no matched pair.

KTO’s insight[86]: Use prospect theory (from behavioral economics). Humans feel losses more strongly than gains. A “thumbs down” should produce a stronger gradient than a “thumbs up.”

Loss Function

\boxed{\mathcal{L}_{\text{KTO}}=\mathbb{E}_{y_{w}}\left[\lambda_{w}(1-v(x,y_{w}))\right]+\mathbb{E}_{y_{l}}\left[\lambda_{l}\cdot v(x,y_{l})\right]}

(8.1)

where $v(x,y)=\sigma\left(\beta\log\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}-z_{\text{ref}}\right)$ , and $z_{\text{ref}}$ is the expected KL divergence (a running baseline).

TRL Implementation

The following shows a minimal working example using HuggingFace TRL.

from trl import KTOConfig, KTOTrainer
 
# Dataset format: {"prompt": str, "completion": str, "label": bool}
# label=True for desirable, label=False for undesirable
kto_dataset = [
    {"prompt": "What's 2+2?", "completion": "The answer is 4.", "label": True},
    {"prompt": "What's 2+2?", "completion": "It might be 5.", "label": False},
]
 
kto_config = KTOConfig(
    output_dir="./kto_output",
    beta=0.1,
    desirable_weight=1.0,        # Weight for good examples
    undesirable_weight=1.0,      # Weight for bad examples (increase for loss aversion)
    learning_rate=5e-7,
    max_length=2048,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    num_train_epochs=1,
    bf16=True,
)
 
trainer = KTOTrainer(
    model=model,
    ref_model=ref_model,  # Or None with LoRA
    args=kto_config,
    train_dataset=kto_dataset,
    tokenizer=tokenizer,
)
trainer.train()

When to Choose KTO

You have binary feedback but not matched pairs
Production thumbs-up/down data at scale
One class dominates (e.g., 90% good, 10% bad) — KTO handles imbalance better
Rapid iteration with noisy labels (more robust than DPO to noise)

IPO — Identity Preference Optimization

Motivation

DPO has a degenerate solution: it can achieve zero loss by making the margin between chosen and rejected infinitely large. In practice, this means DPO overfits — pushing chosen probability to 1 and rejected to 0, memorizing training data.

IPO’s fix[14]: Instead of log-sigmoid (which saturates), use a squared loss that targets a specific margin. The loss is minimized at a finite gap, not at infinity.

Loss Function

\boxed{\mathcal{L}_{\text{IPO}}=\mathbb{E}\left[\left(\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|x)}-\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)}-\frac{1}{2\beta}\right)^{2}\right]}

(8.2)

TRL Implementation

The following shows a minimal working example using HuggingFace TRL.

from trl import DPOConfig, DPOTrainer
 
# IPO is implemented as a DPO loss_type variant in TRL
ipo_config = DPOConfig(
    output_dir="./ipo_output",
    beta=0.1,
    loss_type="ipo",             # The key difference!
    learning_rate=5e-7,
    max_length=2048,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    bf16=True,
    num_train_epochs=1,
)
 
trainer = DPOTrainer(
    model=model, ref_model=None, args=ipo_config,
    train_dataset=pref_dataset, tokenizer=tokenizer, peft_config=lora_config,
)
trainer.train()

When to Choose IPO over DPO

Noisy preference data (crowdsourced, AI-judged with errors)
Observing DPO overfitting (train loss $\to$ 0 but eval degrades)
Want more conservative, robust alignment
Multiple epochs needed (DPO degrades after epoch 1; IPO is more stable)

ORPO — Odds Ratio Preference Optimization

Motivation

All methods so far need a reference model — either as a separate copy (doubles memory) or implicitly via LoRA. ORPO [141] eliminates the reference entirely by combining SFT and preference alignment in a single loss.

Key insight: Use the odds ratio of generating chosen vs rejected as the preference signal. The SFT component naturally prevents collapse (no need for KL regularization).

Loss Function

\boxed{\mathcal{L}_{\text{ORPO}}=\underbrace{\mathcal{L}_{\text{SFT}}(y_{w})}_{\text{standard NLL on chosen}}-\lambda\cdot\underbrace{\log\sigma\left(\log\frac{\text{odds}_{\theta}(y_{w}|x)}{\text{odds}_{\theta}(y_{l}|x)}\right)}_{\text{preference alignment via odds ratio}}}

(8.3)

where $\text{odds}_{\theta}(y|x)=\frac{P_{\theta}(y|x)}{1-P_{\theta}(y|x)}$ .

TRL Implementation

The following shows a minimal working example using HuggingFace TRL.

from trl import ORPOConfig, ORPOTrainer
 
orpo_config = ORPOConfig(
    output_dir="./orpo_output",
    beta=0.1,                    # Odds ratio weight (lambda)
    learning_rate=5e-7,
    max_length=2048,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    bf16=True,
    num_train_epochs=1,
    gradient_checkpointing=True,
)
 
trainer = ORPOTrainer(
    model=model,                 # No ref_model needed!
    args=orpo_config,
    train_dataset=pref_dataset,  # Same format as DPO: prompt/chosen/rejected
    tokenizer=tokenizer,
    peft_config=lora_config,
)
trainer.train()

When to Choose ORPO

Memory-constrained: can’t afford reference model copy (saves 70–140GB for 70B)
Starting from base model (not SFT’d yet) — ORPO does SFT simultaneously
Want simplest possible pipeline: one model, one loss, one training run
Good preference data available from the start

Best-of-N Sampling (Rejection Sampling)

Motivation

Sometimes the simplest approach wins. Best-of-N [262] requires no training at all during the RL phase — just generate multiple candidates and pick the best one.

Algorithm

For each prompt, generate $N$ responses from the policy (typically $N=4$ – $64$ )
Score all responses with a reward model
Select the highest-scoring response
(Optional) Use selected responses as SFT data for the next iteration

\boxed{\text{Best-of-N response}:\quad y^{*}=\arg\max_{y_{i}\sim\pi_{\theta}(\cdot|x)}r_{\phi}(x,y_{i})}

(8.4)

TRL Implementation

The following shows a minimal working example using HuggingFace TRL.

from transformers import pipeline
import numpy as np
 
# Inference-time Best-of-N (manual implementation)
gen_pipeline = pipeline("text-generation", model=model, tokenizer=tokenizer)
 
def best_of_n(prompt, n=16, temperature=0.8):
    """Generate N candidates and return the highest-reward one."""
    candidates = gen_pipeline(
        prompt, num_return_sequences=n,
        temperature=temperature, do_sample=True, max_new_tokens=512,
    )
    scores = [reward_model.score(prompt, c["generated_text"]) for c in candidates]
    return candidates[np.argmax(scores)]["generated_text"]
 
best_response = best_of_n(prompt, n=16)
 
# Training: Rejection Sampling Fine-Tuning (RFT)
from trl import SFTConfig, SFTTrainer
 
# Step 1: Generate and filter
all_responses = []
for prompt in prompts:
    candidates = [generate(prompt, temp=0.9) for _ in range(16)]
    scores = [reward_model.score(prompt, c) for c in candidates]
    best_idx = np.argmax(scores)
    if scores[best_idx] > threshold:  # Quality gate
        all_responses.append({"prompt": prompt, "completion": candidates[best_idx]})
 
# Step 2: SFT on best responses
sft_config = SFTConfig(output_dir="./rft_output", learning_rate=2e-5, num_train_epochs=2, max_seq_length=2048)
trainer = SFTTrainer(model=model, args=sft_config, train_dataset=all_responses, tokenizer=tokenizer)
trainer.train()
# Step 3: Repeat from Step 1 with updated model (iterative RFT)

Scaling Laws for Best-of-N

$N$ Quality GainCostNotes1Baseline $1\times$ Standard sampling4+5–8% win-rate $4\times$ Minimum useful. Good cost/quality ratio16+10–15% win-rate $16\times$ Strong. Often matches PPO quality64+15–20% win-rate $64\times$ Diminishing returns start256+18–22% win-rate $256\times$ Only for critical applications

Summary: Choosing an Alignment Method

We have now surveyed the full landscape of preference optimization and RL-based alignment methods. This section consolidates the key trade-offs into a single reference to help practitioners select the right approach for their constraints.

Method	Models	Data	Compute	Stability	Best For
PPO	4	Online (gen)	Very high	Low	Max quality, complex reasoning
GRPO	2 (no critic)	Online (gen)	High	Medium	Math/code (verifiable rewards)
DPO	2	Offline pairs	Low	High	Style/safety, limited compute
Online DPO	3	Online (gen)	Medium	Medium-High	DPO without distribution shift
KTO	2	Unpaired binary	Low	High	Production feedback, thumbs up/down
IPO	2	Offline pairs	Low	Very high	Noisy labels, anti-overfitting
ORPO	1	Offline pairs	Very low	High	Memory-limited, SFT+align combined
Best-of-N	1+RM	Online (gen)	Medium	Perfect	Strong baseline, data generation

Table 8.1:Cross-method comparison of alignment approaches.

Figure 8.1: Approximate quality vs. compute frontier. Methods above the SFT ceiling line improve beyond what supervised — Figure 8.1:Approximate quality vs. compute frontier. Methods above the SFT ceiling line improve beyond what supervised fine-tuning alone achieves. Position is illustrative and model-dependent.