Chapter 14 LLM Evaluation

Evaluation is the backbone of any rigorous machine learning pipeline, yet it is perhaps the most underappreciated component in the development of large language models. Unlike classical supervised learning, where a held-out test set with ground-truth labels provides a clean signal, evaluating LLMs requires grappling with open-ended generation, subjective quality judgments, multi-step reasoning chains, and the ever-present risk of benchmark contamination. This section provides a systematic treatment of the evaluation landscape: from the taxonomy of evaluation types and the mechanics of human annotation, through the mathematics of ranking metrics and the practicalities of LLM-as-judge, to the pitfalls that silently corrupt evaluation pipelines.

Evaluation Scheme Design

Before collecting a single data point, practitioners must decide what to measure and how to measure it. A principled taxonomy prevents the common mistake of choosing metrics by convenience rather than by alignment with the deployment objective.

Taxonomy of Evaluation Types

Intrinsic evaluation measures properties of the model output in isolation, without reference to a downstream application. Perplexity on a held-out corpus, BLEU score against reference translations, and pass@$k$ on coding benchmarks are all intrinsic. Extrinsic evaluation measures the impact of the model on a real-world task or system: does integrating the LLM into a customer-service pipeline reduce ticket escalation rates? Does the coding assistant increase developer velocity?

Automatic evaluation uses deterministic functions (BLEU, exact match) or learned models (BERTScore, LLM-as-judge) to score outputs without human involvement. Human evaluation involves annotators rating or ranking model outputs. Table 表 14.1 summarises the trade-offs.

Reference-based metrics (BLEU, ROUGE, BERTScore) compare model output to one or more gold-standard references. Reference-free metrics (perplexity, LLM-as-judge, human preference) assess quality without a reference. Reference-free approaches are essential when the output space is too large for exhaustive reference collection, as in open-ended dialogue.

When to Use What

A useful decision framework:

• If the task has a clear correct answer (math, code, factual QA): use exact match or execution-based metrics.
• If the task is open-ended but has reference outputs: use reference-based metrics as a lower bound, supplement with LLM-as-judge.
• If the task is subjective (helpfulness, tone, creativity): use human evaluation or a well-calibrated LLM judge.
• If the task involves multi-step agent behaviour: use task success rate and trajectory efficiency (Section 14.6 Metrics for Agentic Tasks).

Data Collection for Evaluation

High-quality evaluation data is the foundation of trustworthy benchmarks. This section covers the design of human annotation pipelines, statistical measures of annotation quality, and the choice between crowdsourcing and expert annotation.

Human Annotation Pipelines

A robust annotation pipeline consists of five stages:

1. Task definition. Specify the annotation task precisely: what is being rated, on what scale, and with what criteria. Ambiguity at this stage propagates into noisy labels.
2. Guideline development. Write annotation guidelines with worked examples covering edge cases. Iterate with a small pilot group before full deployment.
3. Annotator recruitment and training. Select annotators with appropriate background knowledge. Conduct a calibration session where annotators label the same examples and discuss disagreements.
4. Quality control. Embed gold-standard examples with known labels into the annotation queue. Flag annotators whose accuracy on gold examples falls below a threshold.
5. Aggregation. Combine multiple annotations per item using majority vote, averaging, or a probabilistic model (e.g., Dawid–Skene).

Inter-Annotator Agreement

Raw agreement (fraction of items where all annotators agree) is an inadequate measure because it does not account for chance agreement. Two standard chance-corrected measures are Cohen’s $\kappa$[60] (two annotators) and Fleiss’ $\kappa$[92] (multiple annotators).

Given two annotators labelling $N$ items into $k$ categories, let $p_o$ be the observed agreement and $p_e$ be the expected agreement under independence:

where

and

with $p_{jc}$ being the proportion of items assigned to category $c$ by annotator $j$. Cohen’s $\kappa$ ranges from $-1$ (perfect disagreement) through $0$ (chance agreement) to $1$ (perfect agreement). Values above $0.6$ are generally considered acceptable; above $0.8$ is strong agreement.

For $n$ annotators labelling $N$ items into $k$ categories, let $n_{ij}$ be the number of annotators who assigned item $i$ to category $j$. Define:

Annotation Guideline Design

Effective annotation guidelines share several properties:

• Operationalised criteria. Replace vague terms like “helpful” with concrete, observable behaviours: “The response directly addresses the user’s question and provides all information needed to complete the stated task.”
• Worked examples. Provide at least two examples per rating level, including borderline cases.
• Decision trees. For complex tasks, a flowchart that guides annotators through a sequence of binary decisions reduces cognitive load and improves consistency.
• Explicit scope. State what annotators should not consider (e.g., “Do not penalise for stylistic preferences; focus only on factual accuracy”).

Crowdsourcing vs. Expert Annotation

For safety-critical evaluation (e.g., detecting harmful outputs, evaluating medical advice), expert annotation is non-negotiable. For large-scale preference collection (e.g., building a reward model training set), crowdsourcing with rigorous quality control is often the only feasible option.

Synthetic Data Generation for Evaluation

Human annotation is expensive and slow. Synthetic data generation uses LLMs themselves to produce evaluation data at scale. This section covers the major paradigms.

LLM-as-Judge for Calibration

When using an LLM to generate evaluation labels, calibration is essential: the judge’s scores must be aligned with human judgments. Let $h_i\in [0,1]$ be the human preference score for item $i$ and ${\widehat{h}}_i$ be the judge’s predicted score. Calibration error is measured by the Expected Calibration Error (ECE) [124]:

where $B_b$ is the $b$-th confidence bin, $\text{acc}(B_b)$ is the fraction of items in the bin where the judge agrees with humans, and $\text{conf}(B_b)$ is the mean judge confidence in that bin.

A well-calibrated judge satisfies $𝔼[{\widehat{h}}_i\mid {\widehat{h}}_i=p]=p$ for all $p\in [0,1]$. Calibration can be improved by temperature scaling: replacing the judge’s raw logit $z$ with $z/T$ where $T$ is tuned on a held-out calibration set to minimise negative log-likelihood.

Self-Instruct

Self-Instruct [371] bootstraps instruction-following data from a seed set of human-written tasks. The algorithm:

1. Maintain a task pool initialised with $175$ seed tasks.
2. Sample $8$ tasks from the pool; use them as few-shot examples to prompt the LLM to generate new tasks.
3. Filter generated tasks: remove near-duplicates (ROUGE-L similarity $\phantom{}>0.7$ with any existing task), classify as classification vs. non-classification, and generate input–output instances.
4. Add accepted tasks to the pool.
5. Repeat until the desired pool size is reached.

Evol-Instruct

Evol-Instruct [398] evolves a seed instruction set by iteratively rewriting instructions to be more complex or diverse. Two evolution operators are applied:

• In-depth evolution: Add constraints, increase reasoning steps, concretise abstractions, deepen domain knowledge requirements.
• In-breadth evolution: Generate a new instruction on a related but different topic, increasing topic diversity.

An instruction is accepted if it passes an elimination filter: the evolved instruction must not be a simple copy, must not contain “I’m sorry” or similar refusals, and must not be shorter than the original.

Constitutional AI Data Generation

Constitutional AI (CAI) [16] generates preference data by having the model critique and revise its own outputs according to a set of principles (the “constitution”). The pipeline:

1. Supervised learning phase: Sample a harmful prompt, generate an initial response, then prompt the model to critique the response according to a constitutional principle and revise it. Use the revised response as a supervised fine-tuning target.
2. RL phase: Generate pairs of responses (original vs. revised), use the model to label which is more constitutional, and train a preference model on these labels. Use the preference model as a reward signal for RLHF.

This approach generates preference data without human labelling of harmful content, reducing annotator exposure to distressing material.

Distillation for Evaluation Data

A powerful teacher model (e.g., GPT-4) can generate high-quality evaluation data for training a smaller judge model. The distillation pipeline:

1. Collect a diverse set of prompts and model responses.
2. Use the teacher to generate detailed judgments (scores + rationales).
3. Fine-tune a smaller model on (prompt, response, judgment) triples.
4. Validate the student judge against held-out human annotations.

Arena-Style Pairwise Generation

Chatbot Arena [433] generates evaluation data through a crowdsourced battle platform where users submit prompts and vote on which of two anonymised model responses they prefer. This produces a large-scale, naturally diverse dataset of pairwise preferences. The key design choices:

• Anonymisation: Model identities are hidden to prevent brand bias.
• User-submitted prompts: Ensures prompt diversity and real-world relevance.
• Tie handling: Users can declare a tie or indicate that both responses are bad.
• Deduplication: Near-duplicate prompts are filtered to prevent over-representation of common queries.

Metrics for Ranking Tasks

When the goal is to rank models by quality, pairwise comparison data is more reliable than absolute scores. This section derives the major ranking systems used in LLM evaluation.

ELO Rating System

The ELO system [84], originally developed for chess, assigns each player (model) a scalar rating $R$ such that the expected score of player $A$ against player $B$ is:

The ELO model assumes that each player’s performance on a given game is a random variable drawn from a logistic distribution centred at their rating. The probability that $A$ beats $B$ is:

where $s=400/\ln (10)\approx 173.7$ is a scale parameter chosen so that a 400-point difference corresponds to a $10:1$ odds ratio. After each game with outcome $S_A\in \{0,0.5,1\}$ (loss, draw, win), ratings are updated:

where $K$ is the $K$-factor controlling the learning rate. In Chatbot Arena, $K=4$ is used.

Because ELO ratings depend on the order in which games are processed, confidence intervals are computed by bootstrap resampling: resample the battle log with replacement $B=1000$ times, recompute ELO ratings from scratch for each resample, and report the 2.5th and 97.5th percentiles as the 95% confidence interval.

Bradley–Terry Model

The Bradley–Terry (BT) model [29] is a maximum-likelihood alternative to ELO. Given $n$ models with strength parameters ${\beta }_1,\mathrm{\dots },{\beta }_n>0$, the probability that model $i$ beats model $j$ is:

Given a set of pairwise outcomes ${\{(i_k,j_k,y_k)\}}_{k=1}^M$ where $y_k=1$ if $i_k$ beats $j_k$ and $y_k=0$ otherwise, the log-likelihood is:

The MLE $\widehat{𝜷}$ is found by iterative scaling or gradient ascent. The BT model is identifiable only up to a multiplicative constant; a common normalisation is ${\sum }_i\log {\beta }_i=0$. Working in log-space with ${\theta }_i=\log {\beta }_i$ gives:

which is equivalent to a logistic regression with item-specific intercepts. The BT model is preferred over ELO when the full battle history is available, as it uses all data simultaneously rather than processing games sequentially.

TrueSkill

TrueSkill [138] is a Bayesian skill rating system that models each player’s skill as a Gaussian random variable $s_i\sim 𝒩({\mu }_i,{\sigma }_i^2)$. The performance of player $i$ in a game is $p_i=s_i+{ϵ}_i$ where ${ϵ}_i\sim 𝒩(0,{\beta }^2)$ is game-specific noise. Player $i$ beats player $j$ if $p_i>p_j$.

The posterior update after observing $i\succ j$ is computed via expectation propagation (EP). The key update equations for the winner are:

where $c=\sqrt{2{\beta }^2+{\sigma }_i^2+{\sigma }_j^2}$, and $v(t)=\varphi (t)/\mathrm{\Phi }(t)$, $w(t)=v(t)(v(t)+t)$ are the truncated Gaussian correction factors ($\varphi$ and $\mathrm{\Phi }$ are the standard normal PDF and CDF). TrueSkill’s uncertainty estimate ${\sigma }_i$ is particularly useful for identifying models that need more evaluation data.

Win Rate with Confidence Intervals

The simplest ranking metric is the win rate: the fraction of pairwise comparisons in which model $A$ is preferred. Given $n$ comparisons with $w$ wins, the win rate is $\widehat{p}=w/n$. A Wilson score confidence interval [380] is preferred over the naive Wald interval because it has better coverage near $p=0$ and $p=1$:

where $z=1.96$ for a 95% interval. For multi-way comparisons, win rate should be computed against a fixed baseline model to ensure comparability.

Chatbot Arena Methodology

Chatbot Arena [433] combines the above elements into a production-scale evaluation system:

1. Users submit prompts and receive responses from two anonymised models.
2. Users vote for the preferred response (or declare a tie).
3. Votes are aggregated using the BT model to produce a leaderboard.
4. Bootstrap confidence intervals are reported for each model’s score.
5. Models with overlapping confidence intervals are considered statistically indistinguishable.

As of 2024, Chatbot Arena has collected over one million human preference votes, making it the largest publicly available LLM preference dataset.

Metrics for Generation Tasks

Generation metrics quantify the quality of model outputs for tasks with reference answers or well-defined correctness criteria.

BLEU

BLEU (Bilingual Evaluation Understudy) [283] measures $n$-gram precision between a hypothesis $h$ and one or more references $ℛ$:

where $p_n$ is the modified $n$-gram precision, $w_n=1/N$ are uniform weights, and BP is the brevity penalty:

with $|r|$ being the length of the closest reference. Modified $n$-gram precision clips each $n$-gram count to its maximum count in any reference:

ROUGE

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) [217] is a family of recall-oriented metrics designed for summarisation:

where LCS denotes the longest common subsequence. ROUGE-1 and ROUGE-2 measure unigram and bigram recall; ROUGE-L captures sentence-level structure. The F-measure variant balances precision and recall:

with $\beta =1$ for equal weighting.

BERTScore

BERTScore [426] computes token-level similarity using contextual embeddings from a pre-trained BERT model. Given hypothesis tokens $\widehat{𝐱}=⟨{\widehat{x}}_1,\mathrm{\dots },{\widehat{x}}_m⟩$ and reference tokens $𝐱=⟨x_1,\mathrm{\dots },x_n⟩$ with embeddings ${\widehat{𝐞}}_i$ and ${𝐞}_j$:

BERTScore correlates better with human judgments than BLEU and ROUGE, particularly for paraphrases and semantically equivalent but lexically different outputs. Importance weighting using inverse document frequency (IDF) further improves correlation:

METEOR

METEOR [19] addresses BLEU’s recall blindness by computing an F-score over unigram matches, with additional modules for stemming and synonym matching:

where $F_{\text{mean}}=\frac{10PR}{R+9P}$ (recall-weighted harmonic mean) and the fragmentation penalty $\text{Pen}=0.5\cdot {(c/u_m)}^3$ penalises non-contiguous matches ($c$ = number of chunks, $u_m$ = number of matched unigrams).

Perplexity

Perplexity measures how well a language model predicts a held-out text sequence $w_1,w_2,\mathrm{\dots },w_T$:

Lower perplexity indicates better predictive performance. Perplexity is useful for comparing models on the same tokenisation and test set, but is not directly comparable across models with different vocabularies or tokenisers. For evaluation purposes, perplexity is most useful as a sanity check and for detecting distribution shift.

Pass@k for Code

For code generation, functional correctness is measured by executing generated code against test cases. The pass@$k$ metric [47] estimates the probability that at least one of $k$ generated samples passes all tests:

where $n$ is the total number of samples generated per problem and $c$ is the number that pass. This unbiased estimator avoids the high variance of the naive estimator (which samples exactly $k$ solutions and checks if any pass). In practice, $n=200$ samples are generated and pass@1, pass@10, pass@100 are reported.

Exact Match and F1

For extractive question answering (e.g., SQuAD), two standard metrics are:

• Exact Match (EM): Binary indicator of whether the predicted answer string exactly matches any gold answer after normalisation (lowercasing, removing articles and punctuation).
• Token-level F1: Treats prediction and gold answer as bags of tokens and computes the F1 score:$$F1=\frac{2\cdot |\text{pred}\cap \text{gold}|}{|\text{pred}|+|\text{gold}|}$$(14.30)

For multi-answer settings, the maximum F1 over all gold answers is reported.

Metrics for Agentic Tasks

Agentic LLMs operate in environments, take sequences of actions, and must complete multi-step tasks. Standard generation metrics are insufficient; agentic evaluation requires metrics that capture task completion, efficiency, and the quality of intermediate steps.

Task Success Rate

The primary metric for agentic tasks is the task success rate (TSR): the fraction of tasks for which the agent achieves the specified goal state:

Goal achievement is typically verified by a deterministic oracle (e.g., checking database state, file system state, or test case execution). For tasks with partial credit, a graded success metric can be defined:

Trajectory Efficiency

A successful agent should complete tasks with minimal unnecessary actions. Trajectory efficiency measures the ratio of the optimal trajectory length to the agent’s actual trajectory length:

where $L^{\ast }$ is the length of the shortest successful trajectory (computed by an oracle or human expert) and $L_{\text{agent}}$ is the number of actions taken by the agent. $\eta \in (0,1]$ with $\eta =1$ indicating optimal efficiency. For failed trajectories, $\eta =0$.

A complementary metric is the redundancy rate: the fraction of agent actions that are not present in any optimal trajectory.

Tool-Use Accuracy

For agents that invoke external tools (APIs, code interpreters, search engines), tool-use accuracy measures the correctness of tool calls:

A tool call is correct if (a) the correct tool is selected, (b) the arguments are valid, and (c) the call is made at the appropriate point in the trajectory. Partial credit can be assigned for correct tool selection with incorrect arguments.

Multi-Step Reasoning Accuracy

For tasks requiring chains of reasoning (e.g., multi-hop QA, mathematical problem solving), step-level accuracy measures the fraction of reasoning steps that are correct:

where $S_{\tau }$ is the set of reasoning steps in trajectory $\tau$. Step correctness can be verified by a process reward model (PRM) or by human annotation.

SWE-bench Methodology

SWE-bench [169] evaluates LLMs on real-world software engineering tasks: given a GitHub issue description and the repository codebase, the model must generate a patch that resolves the issue. Evaluation proceeds as follows:

1. The model is given the issue description and relevant code context.
2. The model generates a patch (unified diff format).
3. The patch is applied to the repository.
4. The repository’s test suite is executed; the task is successful if all tests pass.

The primary metric is % Resolved: the fraction of issues for which the generated patch passes all tests. SWE-bench Verified is a curated subset of 500 problems verified by human annotators to be solvable and unambiguous. SWE-bench Lite is a 300-problem subset designed for faster evaluation.

WebArena Methodology

WebArena [440] evaluates agents on realistic web navigation tasks in a sandboxed browser environment. The benchmark includes 812 tasks across five web applications (e-commerce, social forum, collaborative development, content management, and maps). Evaluation:

• Functional evaluation: The task outcome is verified by checking the application state (e.g., “Was the item added to the cart?”, “Was the post created?”).
• URL-based evaluation: For navigation tasks, the final URL is compared to the expected URL.
• Program-based evaluation: A custom evaluator script checks complex conditions (e.g., “Is the price less than $50?”).

The primary metric is task success rate. Human performance is approximately 78%; state-of-the-art agents achieve approximately 35–45%.

LLM-as-Judge

LLM-as-judge [433] uses a capable LLM to evaluate the outputs of other (or the same) LLMs. This approach scales to large evaluation sets without human annotation and can provide detailed rationales for its judgments.

Setup and Prompt Templates

The judge is presented with a prompt, one or more model responses, and an evaluation rubric. Three common formats:

The judge assigns an absolute score to a single response:

The judge compares two responses and selects the better one:

The judge is provided with a reference answer and rates the response relative to it. This is particularly useful for factual tasks where the judge may not have reliable knowledge.

Position Bias Mitigation

LLM judges exhibit position bias: a systematic preference for the response appearing in a particular position (first or last). This bias can be as large as 10–15 percentage points. Mitigation strategies:

1. Swap augmentation: Evaluate each pair in both orders (A vs. B and B vs. A). A consistent judgment is accepted; an inconsistent judgment is recorded as a tie.
2. Calibration prompting: Explicitly instruct the judge: “Your evaluation should not be influenced by the order in which the responses are presented.”
3. Ensemble judging: Use multiple judges with different position orderings and aggregate their verdicts.
4. Chain-of-thought forcing: Require the judge to produce a detailed rationale before the verdict, which reduces reliance on superficial positional cues.

Multi-Judge Panels

A single judge may have systematic biases. A panel of judges from different model families provides more robust evaluations. Given $J$ judges with verdicts $v_1,\mathrm{\dots },v_J\in \{A,B,\text{tie}\}$, the panel verdict is determined by majority vote. The panel agreement rate is:

For a three-judge panel, a unanimous verdict (all three agree) is treated as high-confidence; a 2–1 split as medium-confidence; and a three-way tie as low-confidence.

Agreement Metrics for LLM Judges

To validate an LLM judge, its verdicts are compared to human annotations on a held-out set. Key metrics:

• Agreement rate: Fraction of items where judge and human agree.
• Cohen’s $\kappa$: Chance-corrected agreement (Equation ).
• Spearman’s $\rho$: Rank correlation between judge scores and human scores, appropriate for ordinal ratings.
• Kendall’s $\tau$: Alternative rank correlation that is more robust to ties.

A judge is considered reliable if it achieves $\kappa >0.6$ and agreement rate $\phantom{}>80\%$ with human annotators on a representative sample.

G-Eval Framework

G-Eval [232] is a structured framework for LLM-based evaluation that uses chain-of-thought prompting and token probability weighting to produce more reliable scores. The framework:

1. Generate evaluation steps: Prompt the LLM to generate a detailed rubric for the evaluation task (e.g., “List the steps you would take to evaluate the coherence of a summary”).
2. Score with probability weighting: For each score value $s\in \{1,2,3,4,5\}$, obtain the log-probability $\log P_{\theta }(s\mid \text{prompt, steps, response})$ from the judge model. The final score is the probability-weighted average:$$\text{G-Eval score}=\sum _{s=1}^{5}s\cdot \frac{e^{\log P_{\theta }(s)}}{{\sum }_{s^{\prime }=1}^5{e}^{\log P_{\theta }(s^{\prime })}}$$(14.37)
3. Normalise: Map the score to $[0,1]$ by dividing by the maximum score.

G-Eval achieves higher correlation with human judgments than direct prompting, particularly for nuanced dimensions like coherence and consistency, because the probability weighting captures the judge’s uncertainty rather than forcing a discrete choice.

Evaluation Pitfalls

Even carefully designed evaluation pipelines can produce misleading results. This section catalogues the most common failure modes.

Benchmark Contamination

Benchmark contamination occurs when evaluation data appears in the model’s training set, either directly (verbatim inclusion) or indirectly (paraphrased or semantically similar content). Contaminated models achieve inflated scores that do not reflect true generalisation ability.

• $n$-gram overlap: Compute the fraction of evaluation examples with high $n$-gram overlap (e.g., ROUGE-L $\phantom{}>0.8$) with the training corpus.
• Membership inference: Use a membership inference attack to estimate the probability that each evaluation example was in the training set.
• Canary strings: Embed unique, randomly generated strings in evaluation examples and check if the model can complete them.
• Temporal holdout: Use evaluation data created after the model’s training cutoff date.

• Maintain a private test set that is never released publicly.
• Regularly refresh benchmarks with new examples.
• Report training data cutoff dates and decontamination procedures.

Overfitting to Benchmarks

Even without direct contamination, models can be implicitly optimised for specific benchmarks through repeated evaluation and hyperparameter tuning. This is a form of adaptive overfitting: the benchmark leaks information into model development decisions.

Goodhart’s Law in Evaluation

Goodhart’s Law states: “When a measure becomes a target, it ceases to be a good measure.”[112] In LLM evaluation, this manifests in several ways:

• Reward hacking: Models trained with RLHF learn to exploit the reward model rather than genuinely improving. A model may learn to produce verbose, confident-sounding responses that score highly on the reward model but are factually incorrect.
• Metric gaming: Models fine-tuned to maximise BLEU or ROUGE may produce outputs that score well on these metrics but are less useful to humans.
• Judge gaming: Models trained with LLM-as-judge feedback may learn the judge’s biases (e.g., verbosity bias) rather than genuinely improving quality.

Additional Pitfalls

LLM performance can vary dramatically with small changes to the evaluation prompt (e.g., adding “Think step by step” or changing the answer format). Always report the exact prompt used and consider evaluating across multiple prompt variants.

Averaging scores across tasks with different difficulty levels and score distributions can produce misleading aggregate metrics. A model that excels at easy tasks but fails at hard tasks may have the same average score as a model with uniform performance.

Human evaluators are not a random sample of end users. Annotators on crowdsourcing platforms may have different preferences, cultural backgrounds, and domain knowledge than the target user population.

Evaluation prompts are often shorter, cleaner, and more well-formed than real user queries. A model that performs well on benchmark prompts may degrade significantly on the noisy, ambiguous, multi-turn conversations that occur in production.