Chapter 10 SFT Best Practices and Techniques

Supervised Fine-Tuning (SFT) is the foundation of the RLHF pipeline. The quality of the SFT model determines the ceiling of what RL can achieve: RL can refine and improve a behaviour, but it cannot reliably introduce a behaviour that is entirely absent from the SFT model. This section covers the key techniques for effective SFT.

Sequence Packing for Efficiency

Sequence packing concatenates multiple short examples into a single sequence of length max_seq_length, separated by EOS tokens. The attention mask ensures that tokens from different examples do not attend to each other:

Sort examples by length (optional, improves packing efficiency).
Greedily pack examples into bins of size max_seq_length.
Use a block-diagonal attention mask to prevent cross-example attention.
Compute loss only on non-padding tokens.

Chat Templates and Formatting

ChatML is the most widely used chat template:

# ChatML format
template = """<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{user_message}<|im_end|>
<|im_start|>assistant
{assistant_message}<|im_end|>"""

Llama 3 uses a different template with special tokens:

# Llama 3 format
template = """<|begin_of_text|><|start_header_id|>system<|end_header_id|>
{system_message}<|eot_id|><|start_header_id|>user<|end_header_id|>
{user_message}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
{assistant_message}<|eot_id|>"""

Completion-Only Masking

Data Mixing Strategies for Multi-Task SFT

Sample from each dataset proportionally to its size:

p_{k}=\frac{N_{k}}{\sum_{j=1}^{K}N_{j}},

where $N_{k}$ is the number of examples in dataset $k$ . This is the default in most frameworks and works well when datasets are of similar quality.

Apply a temperature $T$ to smooth the proportions:

p_{k}\propto N_{k}^{1/T}.

$T=1$ : proportional mixing. $T\to\infty$ : uniform mixing. $T<1$ : over-samples large datasets. $T>1$ : over-samples small datasets.

Weight datasets by estimated quality (e.g., perplexity under a reference model, human quality ratings):

p_{k}\propto N_{k}\cdot q_{k},

where $q_{k}$ is the quality score for dataset $k$ .

When SFT Hurts – Catastrophic Forgetting and Alignment Tax

As LLMs transition through sequential training phases — pre-training $\rightarrow$ continued pre-training $\rightarrow$ SFT $\rightarrow$ RLHF/DPO — performance degradation frequently manifests on standard benchmarks. Two fundamentally distinct phenomena drive these regressions, and confusing them leads to wrong mitigation strategies.

Catastrophic Forgetting (Structural Erasure)

Symptoms:

Complete breakdown on tasks not in fine-tuning data (e.g., model forgets how to do math after SFT on chat data)
Loss of language diversity — model only generates in the narrow style of fine-tuning distribution
Reduced factual accuracy on knowledge not reinforced during fine-tuning
Degraded multilingual ability after English-only SFT

Mechanistic cause — Fisher Information perspective: The Fisher Information Matrix $F$ of Task A identifies which parameters are “important” for $\mathcal{D}_{A}$ :

F=\mathbb{E}_{x\sim\mathcal{D}_{A}}\!\left[\nabla_{\theta}\log\pi_{\theta}(x)\,\nabla_{\theta}\log\pi_{\theta}(x)^{T}\right]

(10.2)

Parameters with high Fisher eigenvalues are critical for Task A. Unconstrained gradient descent on Task B ignores these eigenvalues entirely — $\Delta\theta$ points along $\nabla\mathcal{L}_{B}$ regardless of whether it destroys high-Fisher directions for $\mathcal{L}_{A}$ .

Alignment Tax (Behavioral Constraint)

The alignment tax is a deliberate, expected trade-off: the model’s raw capability (unconstrained generation, maximal reasoning bandwidth) decreases because the policy is constrained to produce safe, well-formatted, preference-aligned outputs.

Mechanism: During DPO/PPO, the policy $\pi_{\theta}$ is penalized for deviating from the reference $\pi_{\text{ref}}$ via KL divergence:

r_{\text{implicit}}(x,y)=\beta\log\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}

(10.3)

This leash constrains the model’s output distribution — it cannot explore high-variance reasoning paths that deviate too far from the reference. The knowledge is not erased; it’s suppressed. The model still “knows” the answer but its distribution is flattened toward safe, generic responses.

Symptoms:

Over-refusal (“I can’t help with that” for benign queries)
Stylistic stiffness — hedge words, excessive caveats, verbose safety disclaimers
Lower scores on raw capability benchmarks (MMLU, HumanEval) while improving on preference benchmarks (MT-Bench, AlpacaEval)
Reduced ability to produce complex, high-entropy outputs (creative writing, novel algorithms)

Comparative Taxonomy

Dimension	Catastrophic Forgetting	Alignment Tax
Intentionality	Unintentional (optimization artifact)	Expected trade-off (incurred deliberately for safety/helpfulness)
Parameter state	Prior knowledge physically overwritten	Latent distributions constrained/truncated
Information	Destroyed: weights no longer encode the capability	Suppressed: knowledge exists but is harder to trigger
Dominant phase	Sequential SFT, domain continued pre-training	Preference optimization (PPO, DPO, KTO, RLHF)
Primary symptom	Complete breakdown of baseline capabilities	Over-refusal, stylistic stiffness, lower raw benchmark scores
Reversibility	Irreversible without retraining from checkpoint	Partially reversible: adjust $\beta$ , system prompt, or fine-tune
Detection	Perplexity on pre-training eval set spikes	Perplexity stable but win-rate on capability benchmarks drops
Scales with model size	Similar across scales	Smaller models pay a larger alignment tax

Table 10.1:Catastrophic Forgetting vs. Alignment Tax — complete comparison.

Mitigation Strategies

For Catastrophic Forgetting:

Data replay: Mix 5–10% of pre-training data into SFT dataset. Ensures gradient updates don’t completely neglect pre-training distribution.
Elastic Weight Consolidation (EWC)[186]: Add regularization $\Omega(\theta)=\frac{\lambda}{2}\sum_{i}F_{i}(\theta_{i}-\theta_{i}^{*})^{2}$ that penalizes changes to parameters with high Fisher information for the original task.
LoRA / Parameter-efficient fine-tuning: Train only low-rank adapters ( $<1\%$ of parameters), leaving base weights completely frozen. This prevents permanent destruction of pre-trained knowledge — you can always remove the adapter and recover the original model. However, while the adapter is active, the combined system $(W_{0}+BA)$ can still exhibit forgetting: the adapter may shift the model’s effective behavior away from old skills. LoRA protects the checkpoint, not the active inference behavior.
Conservative learning rate: Use $1$ – $5\times 10^{-6}$ with few epochs (1–3). Larger rates accelerate forgetting.
Progressive training: Mix distributions gradually, increasing SFT data proportion over time rather than switching abruptly.

For Alignment Tax:

Tune $\beta$ carefully: Lower $\beta$ gives the model more freedom (reduces the tax) but may sacrifice safety. Optimal $\beta\in[0.05,0.3]$ for most settings.
High-quality, diverse SFT data: Part of the alignment tax comes from SFT narrowing the output distribution; broader, more diverse SFT data reduces this component. The RL phase adds further constraint via KL regularization [280].
Conditional alignment: Train the model to be aligned only when a safety flag is active. At inference, disable constraints for benchmarking (research-only technique).
Constitutional AI / RLAIF: Use model-generated feedback to create more nuanced preference data that preserves capability while improving alignment.
Targeted RL budget: Don’t over-train with RL. Monitor capability benchmarks and stop when the tax exceeds acceptable thresholds (typically 2–5% MMLU regression).

Connection to RL – SFT Quality Determines RL Ceiling

SFT data quality: use high-quality, diverse data. A small amount of high-quality data is better than a large amount of low-quality data.
SFT data coverage: ensure the SFT data covers the tasks you want to improve with RL. If a task is not in the SFT data, RL will struggle.
SFT training duration: do not over-train the SFT model. Over-training reduces diversity and makes RL exploration harder.
Warm-up: consider a short SFT warm-up on task-specific data before RL, even if the base model is already instruction-tuned.