Chapter 9 Reward Model Training

Reward models are the bridge between human preferences and the RL training signal. A well-trained reward model is essential for successful RLHF; a poorly trained one leads to reward hacking and misaligned behaviour. This section covers the theoretical foundations, practical training techniques, and architectural choices for reward models.

Bradley-Terry Model – Full Derivation

The Bradley-Terry model [29] is the standard probabilistic framework for pairwise preference learning. Given two responses $y_{1}$ and $y_{2}$ to a prompt $q$ , the model assumes:

P(y_{1}\succ y_{2}\mid q)=\sigma(r(y_{1},q)-r(y_{2},q))=\frac{e^{r(y_{1},q)}}{e^{r(y_{1},q)}+e^{r(y_{2},q)}},

where $r:\mathcal{Y}\times\mathcal{Q}\to\mathbb{R}$ is the scalar reward function and $\sigma$ is the sigmoid function.

Given a dataset $\mathcal{D}=\{(q^{(k)},y_{w}^{(k)},y_{l}^{(k)})\}_{k=1}^{N}$ of preference pairs, the MLE objective is:

\mathcal{L}_{\text{BT}}(\phi)=-\frac{1}{N}\sum_{k=1}^{N}\log\sigma\!\bigl(r_{\phi}(y_{w}^{(k)},q^{(k)})-r_{\phi}(y_{l}^{(k)},q^{(k)})\bigr),

where $r_{\phi}$ is a neural network parameterised by $\phi$ . This is a binary cross-entropy loss where the “positive” class is the preferred response.

A common extension adds a margin $m$ to ensure a minimum gap between winning and losing rewards:

\mathcal{L}_{\text{margin}}=-\frac{1}{N}\sum_{k=1}^{N}\log\sigma\!\bigl(r_{\phi}(y_{w}^{(k)},q^{(k)})-r_{\phi}(y_{l}^{(k)},q^{(k)})-m\bigr).

Reward Model Architectures

The architecture is:

Backbone: a pretrained LLM (e.g., Llama, Mistral) that encodes the prompt-response pair into a sequence of hidden states.
Pooling: extract the hidden state at the last token position (for decoder-only models) or the [CLS] token (for encoder models).
Regression head: a linear layer $W\in\mathbb{R}^{d\times 1}$ that maps the pooled hidden state to a scalar reward.

Reward Model Training Tricks

Raw reward model outputs can have arbitrary scale and offset. Centering the rewards (subtracting the mean) stabilises RL training:

r_{\text{centered}}(y,q)=r_{\phi}(y,q)-\mathbb{E}_{y^{\prime}\sim\pi_{\theta}}[r_{\phi}(y^{\prime},q)].

In TRL, this is implemented via the center_rewards_coefficient parameter, which adds a regularisation term to the reward model loss that penalises non-zero mean rewards.

Reward models are known to exhibit length bias: they tend to assign higher rewards to longer responses, regardless of quality. This can be corrected by:

Length normalisation: divide the reward by the response length.
Length-controlled training: include length as a feature and train the model to be length-invariant.
Calibration: post-hoc regression to remove the length effect.

Adding a margin $m$ to the Bradley-Terry loss ensures the reward model assigns meaningfully different scores to preferred and dispreferred responses:

\mathcal{L}_{\text{margin}}=\max\!\bigl(0,\;m-(r_{w}-r_{l})\bigr).

Process Reward Models vs Outcome Reward Models

Step-level annotations can be generated automatically using:

Monte Carlo rollouts: for each intermediate step, sample multiple completions and use the fraction that reach the correct answer as the step reward.
LLM-as-judge: use a strong LLM to evaluate each step.
Formal verification: for math/code, use a verifier to check each step.

Rule-Based Rewards for RLVR

Reinforcement Learning from Verifiable Rewards (RLVR) uses deterministic, rule-based reward functions instead of learned reward models. This substantially reduces reward hacking (though models can still exploit format tricks, edge cases, or test memorization) and is the approach used in DeepSeek-R1 [69].

Multi-Objective Rewards – Combination Strategies

When training with multiple reward signals, the combination strategy significantly affects the final policy.

Listwise Rank-Based Rewards

While the Bradley-Terry model handles pairwise preferences ( $y_{w}\succ y_{l}$ ), many practical scenarios involve ranking multiple responses simultaneously. Listwise reward models learn from complete orderings, providing richer training signal and enabling better calibration.

The Plackett-Luce (PL) model [291] is the standard extension of Bradley-Terry to full rankings. Given $K$ responses $y_{1},\ldots,y_{K}$ with ranking $\pi$ (where $\pi(1)$ is the best):