Chapter 16 Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation (RAG) [209] has emerged as one of the most practically impactful techniques for deploying large language models in production. Rather than relying solely on knowledge encoded in model weights during training, RAG equips LLMs with a dynamic, updatable external memory—enabling accurate, grounded, and verifiable responses across a wide range of knowledge-intensive tasks.

Motivation and Problem Statement

Parametric vs. Non-Parametric Knowledge

We can formalize the distinction between the two knowledge sources. Let ${ℳ}_{\theta }$ denote a language model with parameters $\theta$, and let $𝒟=\{d_1,d_2,\mathrm{\dots },d_N\}$ be an external document corpus. The probability of generating answer $a$ given query $q$ under each paradigm is:

where $P_{\text{ret}}(d\mid q,𝒟)$ is the retrieval distribution over documents. RAG marginalizes over retrieved evidence, grounding generation in non-parametric knowledge.

When to Use RAG vs. Fine-Tuning vs. Long Context

Core RAG Architecture

A standard RAG system consists of two phases: an offline indexing pipeline that processes and stores documents, and an online retrieval-generation pipeline that serves queries.

Full Pipeline Diagram

Indexing Pipeline

Documents arrive in heterogeneous formats (PDF, HTML, Markdown, DOCX, code). Loaders extract clean text and preserve metadata (source URL, page number, section title, timestamp) that will be stored alongside embeddings for filtering and citation.

Long documents must be split into chunks that fit within the embedding model’s context window (typically 512 tokens) and are semantically coherent. Chunking strategy is one of the highest-impact decisions in RAG system design (see Section 16.4 Chunking Strategies).

Each chunk $c_i$ is encoded into a dense vector ${𝐞}_i=f_{\varphi }(c_i)\in {ℝ}^d$ using an embedding model $f_{\varphi }$. These vectors are stored in a vector database alongside the original text and metadata.

Retrieval

Given a query $q$, the retrieval step encodes it as $𝐪=f_{\varphi }(q)$ and finds the $k$ most similar chunks by cosine similarity:

The top-$k$ chunks ${𝒞}_k=\{c_{(1)},\mathrm{\dots },c_{(k)}\}$ are returned as context.

Generation

Retrieved chunks are injected into a prompt template:

SYSTEM_PROMPT = """You are a helpful assistant. Answer the question using ONLY
the provided context. If the context does not contain enough information,
say so explicitly. Cite your sources using [Doc N] notation."""

def build_rag_prompt(query: str, chunks: list[dict]) -> str:
    context_str = "\n\n".join(
        f"[Doc {i+1}] (Source: {c['source']}, Page: {c.get('page','N/A')})\n{c['text']}"
        for i, c in enumerate(chunks)
    )
    return f"""{SYSTEM_PROMPT}

Context:
{context_str}

Question: {query}

Answer:"""

Listing 8:Standard RAG prompt template

Retrieval Methods

Sparse Retrieval: BM25 and TF-IDF

Sparse retrieval methods represent documents and queries as high-dimensional sparse vectors over the vocabulary. The classic BM25 scoring function [309] for document $d$ given query $q$ with terms $t_1,\mathrm{\dots },t_n$ is:

where $f(t_i,d)$ is term frequency, $|d|$ is document length, avgdl is average document length, and $k_1\in [1.2,2.0]$, $b=0.75$ are tuning parameters.

Dense Retrieval: DPR

Dense Passage Retrieval (DPR) [180] uses two separate BERT-based encoders—a query encoder$E_Q$ and a passage encoder$E_P$—trained with contrastive loss to place relevant query-passage pairs close together in embedding space.

Given a batch of $B$ query-passage pairs ${\{(q_i,p_i^{+})\}}_{i=1}^B$, the contrastive loss treats all other passages in the batch as negatives:

where $\tau$ is a temperature hyperparameter. Hard negatives (passages that are lexically similar but semantically irrelevant) are crucial for training strong retrievers.

At scale, exhaustive search over millions of embeddings is infeasible. FAISS [171] (Facebook AI Similarity Search) provides efficient approximate nearest neighbor (ANN) search using:

• IVF (Inverted File Index): cluster vectors into Voronoi cells; search only nearby cells
• HNSW (Hierarchical Navigable Small World)[248]: graph-based index with $O(\log N)$ search
• PQ (Product Quantization): compress vectors to reduce memory footprint

Hybrid Retrieval with Reciprocal Rank Fusion

Hybrid retrieval combines sparse and dense scores. A simple linear combination is:

However, scores from different systems are not directly comparable. Reciprocal Rank Fusion (RRF)[62] avoids this by operating on ranks rather than scores:

where $ℛ$ is the set of ranked lists (e.g., BM25 ranking and dense ranking), ${\text{rank}}_r(d)$ is the rank of document $d$ in list $r$, and $k=60$ is a smoothing constant that reduces the impact of very high-ranked documents.

Learned Sparse Retrieval: SPLADE and SPLADEv2

SPLADE (Sparse Lexical and Expansion Model) [95] uses a pre-trained masked language model (e.g., BERT/DistilBERT) to produce a sparse vector over the entire vocabulary for each document or query. The key insight: the MLM head already knows which words are semantically related to each position in a text — SPLADE repurposes this knowledge as term importance weights.

Given input text $x=[x_1,\mathrm{\dots },x_n]$:

1. Pass through a transformer encoder to get contextual representations $𝐇\in {ℝ}^{n\times |𝒱|}$ via the MLM head
2. Aggregate across positions and apply a saturating activation:

where ${𝐇}_i[t]$ is the MLM logit for vocabulary token $t$ at input position $i$.

• The $\log (1+\cdot )$ saturation prevents any single term from dominating (similar to TF saturation in BM25)
• The ReLU ensures sparsity — most vocabulary terms get weight zero
• The $\max$ pooling across positions captures the strongest signal for each term from any position in the text
• Expansion: Even tokens not present in the original text can get non-zero weight (e.g., a document about “neural networks” may get weight for “deep learning,” “AI,” “backpropagation”)

Query and document are each mapped to sparse vectors ${𝐰}^q,{𝐰}^d\in {ℝ}^{|𝒱|}$. The relevance score is a simple dot product:

Because both vectors are sparse (typically 20–200 non-zero entries out of 30K vocabulary), this can be computed efficiently using standard inverted indexes (Lucene, Anserini) — no GPU needed at query time.

SPLADE is trained with contrastive learning (in-batch negatives + hard negatives) plus two regularization terms:

The $L_1$ penalties on query and document representations encourage sparsity — without them, the model would learn dense representations that defeat the purpose.

SPLADEv2 [94] introduces several refinements that significantly improve efficiency and effectiveness:

1. Distillation from cross-encoder: Instead of training only on binary relevance labels, SPLADEv2 uses a cross-encoder teacher (e.g., MonoT5 [269]) to provide soft relevance scores. This gives richer training signal:$${ℒ}_{\text{distill}}=\text{KL}(\sigma (s_{\text{student}})\parallel \sigma (s_{\text{teacher}}))$$(16.12)
2. Separate query/document encoders: SPLADEv2 uses different sparsity targets for queries vs. documents. Queries are encouraged to be more sparse (faster lookup) while documents can be slightly denser (pre-computed offline):$${\lambda }_q>{\lambda }_d\text{(e.g., }{\lambda }_q=3\times {10}^{-4},{\lambda }_d=1\times {10}^{-4}\text{)}$$(16.13)
3. FLOPS regularization: Instead of simple $L_1$, SPLADEv2 introduces a FLOPS-aware regularizer that directly penalizes the expected retrieval cost:$${ℒ}_{\text{FLOPS}}=\sum _{t\in 𝒱}{\left({\overline{a}}_t^q\right)}^2+\sum _{t\in 𝒱}{\left({\overline{a}}_t^d\right)}^2$$(16.14)where ${\overline{a}}_t$ is the mean activation for term $t$ across the batch. This penalizes terms that are non-zero for many documents (high posting list length = slow retrieval).
4. Efficient backbone: Uses DistilBERT (66M params) instead of BERT-base (110M), halving encoding time with minimal quality loss.

ColBERT: Late Interaction

ColBERT [183] encodes queries and documents into sets of token-level embeddings and uses a MaxSim operator for scoring:

This late interaction mechanism is more expressive than single-vector bi-encoders while being far faster than cross-encoders, since document embeddings are pre-computed offline.

Both the query encoder $E_Q$ and document encoder $E_D$ are BERT-based models that produce per-token embeddings (not a single [CLS] vector). Each token embedding is projected to a lower dimension (typically 128) via a linear layer:

ColBERT is trained with a pairwise softmax cross-entropy loss over positive and negative passages. Given a query $q$, a positive passage $d^{+}$, and a set of negative passages $\{d_1^{-},\mathrm{\dots },d_N^{-}\}$:

where $s(q,d)$ is the MaxSim score from Equation 16.15. Negatives are sourced from:

• In-batch negatives: Other passages in the same training batch (free, abundant)
• Hard negatives: Passages retrieved by BM25 that are lexically similar but semantically irrelevant (most impactful for quality)
• Distillation negatives (ColBERTv2 [313]): Use a cross-encoder teacher to mine the hardest negatives and distill its scores into ColBERT

At index time, all document token embeddings are pre-computed and stored (with optional compression via residual quantization in ColBERTv2). At query time, only the query tokens are encoded live, and MaxSim is computed against the stored document embeddings. This separation enables:

• Offline document encoding: Encode once, serve many queries
• PLAID indexing[313]: Cluster document embeddings, use centroids for initial candidate retrieval, then compute exact MaxSim only on candidates—reducing latency by 5–10$\times$
• Index size: $|d|\times 128$ floats per document (larger than single-vector methods but compressible to $\sim$2 bytes/dimension with quantization)

Retrieval Method Comparison

Chunking Strategies

Chunking is the process of splitting documents into segments that are (1) small enough to fit in an embedding model’s context window, (2) semantically coherent, and (3) contain enough context to be useful when retrieved in isolation.

Fixed-Size Chunking with Overlap

The simplest strategy: split every $W$ tokens with an overlap of $O$ tokens between consecutive chunks.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,       # tokens per chunk
    chunk_overlap=64,     # overlap to preserve context across boundaries
    length_function=len,
    separators=["\n\n", "\n", ". ", " ", ""]
)
chunks = splitter.split_documents(documents)

Listing 9:Fixed-size chunking with overlap

Overlap formula: For a document of length $L$ tokens, the number of chunks is:

Semantic Chunking

Rather than splitting at fixed intervals, semantic chunking splits at topic boundaries detected by measuring embedding similarity between consecutive sentences:

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

chunker = SemanticChunker(
    embeddings=OpenAIEmbeddings(),
    breakpoint_threshold_type="percentile",  # or "standard_deviation"
    breakpoint_threshold_amount=95,          # split at top 5% dissimilarity
)
chunks = chunker.split_documents(documents)

Listing 10:Semantic chunking via embedding similarity

Document-Structure-Aware Chunking

For structured documents (Markdown, HTML, code), split at natural boundaries:

• Markdown: split at ## headers, preserving section context
• HTML: split at ⟨section⟩, ⟨article⟩, ⟨p⟩ tags
• Code: split at function/class definitions, preserving imports in each chunk
• Tables: keep entire tables as single chunks; never split mid-row

Parent-Child Chunking

A powerful pattern that decouples retrieval granularity from generation context:

1. Index small child chunks (e.g., 128 tokens) for precise retrieval
2. Return large parent chunks (e.g., 512 tokens) to the LLM for richer context

from langchain.retrievers import ParentDocumentRetriever
from langchain.storage import InMemoryStore
from langchain.text_splitter import RecursiveCharacterTextSplitter

parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000)
child_splitter  = RecursiveCharacterTextSplitter(chunk_size=400)

retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=InMemoryStore(),
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,
)
retriever.add_documents(documents)

Listing 11:Parent-child chunking with LangChain

Empirical Guidelines for Chunk Size

Advanced RAG Patterns

Query Transformation

Raw user queries are often ambiguous, too short, or poorly matched to document language. Query transformation techniques improve retrieval before the search step.

Instead of embedding the query directly, generate a hypothetical answer and embed that:

The intuition: a hypothetical answer is in the same linguistic register as real documents, reducing the query-document distribution gap.

For specific questions, first generate a more general “step-back” question, retrieve for both, and combine the contexts. Example: “What is the boiling point of ethanol at 2 atm?” $\to$ step-back: “What factors affect the boiling point of liquids?”

Generate $M$ diverse reformulations of the query, retrieve for each, and union the results:

from langchain.retrievers.multi_query import MultiQueryRetriever
from langchain_openai import ChatOpenAI

retriever = MultiQueryRetriever.from_llm(
    retriever=vectorstore.as_retriever(search_kwargs={"k": 5}),
    llm=ChatOpenAI(temperature=0.7),
    include_original=True,   # also retrieve for original query
)
# Internally generates 3 query variants, retrieves for each, deduplicates
docs = retriever.get_relevant_documents(query)

Listing 12:Multi-query retrieval

Re-Ranking

After initial retrieval of top-$k$ candidates, a cross-encoder re-ranker scores each query-document pair jointly (attending to both simultaneously), producing much more accurate relevance scores at the cost of higher latency:

Cross-encoders cannot be used for first-stage retrieval (no pre-computed document embeddings), but are ideal for re-ranking a small candidate set (typically $k=20$–$100$).

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-large")

def rerank(query: str, docs: list[str], top_n: int = 5) -> list[str]:
    pairs = [(query, doc) for doc in docs]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(scores, docs), reverse=True)
    return [doc for _, doc in ranked[:top_n]]

Listing 13:Cross-encoder re-ranking with BGE

Contextual Compression

Retrieved chunks often contain irrelevant sentences surrounding the relevant passage. Contextual compression uses an LLM to extract only the relevant portions:

from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import LLMChainExtractor

compressor = LLMChainExtractor.from_llm(llm)
compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=vectorstore.as_retriever()
)
compressed_docs = compression_retriever.get_relevant_documents(query)

Listing 14:LLM-based contextual compression

Self-RAG

Self-RAG [13] trains a single model to (1) decide whether to retrieve, (2) generate with or without retrieval, and (3) critique its own output using special reflection tokens:

• [Retrieve]: should the model retrieve additional passages?
• [IsRel]: is the retrieved passage relevant to the query?
• [IsSup]: does the generated statement follow from the retrieved passage?
• [IsUse]: is the overall response useful?

The model is trained end-to-end to predict these tokens alongside the response, enabling fine-grained control over retrieval and self-grading.

CRAG: Corrective RAG

CRAG [401] adds a retrieval evaluator that grades retrieved documents and triggers corrective actions:

1. Retrieve top-$k$ documents
2. Grade each document: Correct / Ambiguous / Incorrect
3. If all documents are incorrect or ambiguous $\to$ fall back to web search
4. If some documents are correct $\to$ use knowledge refinement (strip irrelevant sentences)
5. Generate answer from refined context

Adaptive RAG

Adaptive RAG [162] routes queries to different retrieval strategies based on predicted complexity:

• No retrieval: simple factual queries the model can answer from parameters
• Single-step RAG: standard retrieve-then-generate for moderate queries
• Multi-step RAG: iterative retrieval for complex multi-hop questions

A lightweight classifier trained on query complexity labels routes each incoming query.

Graph RAG

Microsoft’s Graph RAG [82] constructs a knowledge graph from the document corpus and uses community detection to generate hierarchical summaries:

1. Entity extraction: LLM extracts entities and relationships from each chunk
2. Graph construction: build a graph $G=(V,E)$ where nodes are entities and edges are relationships
3. Community detection: apply Leiden algorithm to find communities at multiple resolutions
4. Community summaries: LLM generates a summary for each community
5. Query: for global queries, map-reduce over community summaries; for local queries, use standard vector search

RAG-Fusion

RAG-Fusion [300] generates multiple search queries from the original, retrieves for each, and fuses the ranked lists using RRF (Equation 16.8):

def reciprocal_rank_fusion(ranked_lists: list[list[str]], k: int = 60) -> list[str]:
    """Fuse multiple ranked document lists using RRF."""
    scores: dict[str, float] = {}
    for ranked in ranked_lists:
        for rank, doc_id in enumerate(ranked, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

def rag_fusion(query: str, retriever, llm, n_queries: int = 4) -> str:
    # Step 1: Generate query variants
    variants = generate_query_variants(query, llm, n=n_queries)
    # Step 2: Retrieve for each variant
    all_ranked = [retriever.retrieve(q) for q in [query] + variants]
    # Step 3: Fuse with RRF
    fused_docs = reciprocal_rank_fusion(all_ranked)
    # Step 4: Generate answer
    return generate_answer(query, fused_docs[:5], llm)

Listing 15:RAG-Fusion with RRF

Efficient RAG Decoding: REFRAG

A practical bottleneck of RAG is decoding latency: the retrieved passages concatenated into the LLM context are often long yet sparsely relevant, inflating time-to-first-token (TTFT) and KV-cache memory. REFRAG [221] observes that because retrieved passages are independently sourced (via diversity or deduplication during re-ranking), their attention patterns are block-diagonal—most cross-passage attention is near zero. This sparsity means that the majority of computations over the RAG context during decoding are unnecessary.

REFRAG exploits this structure via a three-phase decoding strategy:

1. Compress: Replace full KV representations of retrieved passages with compact summaries (e.g., mean-pooled keys/values per passage block), drastically reducing memory.
2. Sense: At each decoding step, use lightweight attention over the compressed representations to identify which passage blocks are relevant to the current token.
3. Expand: Reconstruct full KV entries only for the selected blocks, performing exact attention over the sparse active set.

On LLaMA-based models, REFRAG achieves up to $30.85\times$ TTFT speedup (a $3.75\times$ improvement over prior sparse-attention baselines) with no loss in perplexity. It also extends effective context length by $16\times$ under fixed memory budgets. These gains hold across RAG, multi-turn conversation, and long-document summarization tasks.

Agentic RAG

Motivation: Limits of Static RAG

Standard RAG follows a fixed retrieve-then-generate pattern. This fails on:

• Multi-hop questions: “Who founded the company that acquired OpenAI’s main competitor in 2023?” requires chaining multiple retrievals
• Ambiguous queries: the right retrieval strategy depends on what is found
• Heterogeneous sources: different sub-questions require different knowledge bases
• Iterative refinement: initial retrieval may reveal that a different query is needed

Agentic RAG Architecture

Multi-Source Routing

An agentic RAG system can route sub-queries to specialized knowledge sources. The core insight is that different question types demand different retrieval backends—no single index excels at everything.

Consider a financial analyst’s assistant handling four queries:

• “What is our company’s PTO policy?” $\to$Vector DB (internal documents)
• “What did the Fed announce yesterday?” $\to$Web search (real-time)
• “Show Q3 revenue by region” $\to$SQL database (structured data)
• “How does our auth middleware validate tokens?” $\to$Code index (codebase)

A flat retrieve-from-one-index approach either misses the answer or returns irrelevant passages. Routing selects the right tool for the right sub-question before retrieval begins.

Three main approaches, in increasing sophistication:

1. Rule-based routing. Keyword triggers (e.g., SQL keywords $\to$ database, URL patterns $\to$ web). Fast and interpretable but brittle for ambiguous queries.
2. Classifier-based routing. A lightweight model (e.g., a fine-tuned BERT classifier or logistic regression over query embeddings) predicts the best source. Low latency (10 ms) and trainable on routing logs, but requires labeled data.
3. LLM-based routing. The LLM itself decides the source in a structured-output call (see Listing below). Most flexible—handles novel query types and can explain its reasoning—but adds one LLM call of latency.

• Fallback chains: If the primary source returns low-confidence results, try the next-best source.
• Parallel fan-out: For ambiguous queries, retrieve from multiple sources simultaneously and fuse results via Reciprocal Rank Fusion (Table 表 16.2).
• Cost awareness: Web search and API calls may have monetary cost or rate limits; the router should factor these in.
• Observability: Log every routing decision with its reasoning—essential for debugging and retraining.

from enum import Enum
from pydantic import BaseModel

class KnowledgeSource(str, Enum):
    VECTOR_DB   = "vector_db"    # internal documents
    WEB_SEARCH  = "web_search"   # real-time web
    SQL_DB      = "sql_db"       # structured data
    CODE_INDEX  = "code_index"   # codebase
    API         = "api"          # external APIs

class RouteDecision(BaseModel):
    source: KnowledgeSource
    refined_query: str
    reasoning: str

def route_query(query: str, llm) -> RouteDecision:
    """Use LLM to decide which knowledge source to query."""
    prompt = f"""Given the query: "{query}"

Decide which knowledge source to use:
- vector_db: for internal documents, policies, past reports
- web_search: for current events, recent information
- sql_db: for numerical data, statistics, structured records
- code_index: for code examples, API documentation
- api: for real-time data (weather, stock prices, etc.)

Return a JSON with: source, refined_query, reasoning."""

    return llm.with_structured_output(RouteDecision).invoke(prompt)

Listing 16:Multi-source agentic RAG router

Full Agentic RAG Implementation

The previous sections introduced individual components—routing, retrieval, evaluation. A full agentic RAG system orchestrates these as a graph of stateful nodes, where control flow depends on intermediate results. The implementation below uses LangGraph to wire four nodes into a loop:

1. Plan: Decompose the user query into sub-queries (one per information need).
2. Retrieve: Route each sub-query to the appropriate source and fetch documents.
3. Evaluate: Judge whether the accumulated context is sufficient to answer the original query.
4. Generate: Synthesize a final answer with citations from the retrieved documents.

The key design pattern is the conditional loop: after evaluation, the agent either proceeds to generation (if context is sufficient or the iteration budget is exhausted) or loops back to retrieval with refined sub-queries. This mirrors the sense–act–evaluate cycle of an RL agent operating over information-gathering actions.

from typing import TypedDict, Annotated
from langgraph.graph import StateGraph, END
from langgraph.prebuilt import ToolNode
import operator

class AgentState(TypedDict):
    query: str
    sub_queries: list[str]
    retrieved_docs: Annotated[list[dict], operator.add]
    context_sufficient: bool
    answer: str
    iterations: int
    max_iterations: int

def plan_node(state: AgentState) -> AgentState:
    """Decompose query into sub-queries."""
    sub_queries = decompose_query(state["query"])
    return {**state, "sub_queries": sub_queries, "iterations": 0}

def retrieve_node(state: AgentState) -> AgentState:
    """Retrieve documents for current sub-queries."""
    new_docs = []
    for sq in state["sub_queries"]:
        source = route_query(sq)
        docs = retrieve_from_source(sq, source)
        new_docs.extend(docs)
    return {**state, "retrieved_docs": new_docs,
            "iterations": state["iterations"] + 1}

def evaluate_node(state: AgentState) -> AgentState:
    """Evaluate whether retrieved context is sufficient."""
    sufficient = evaluate_context_sufficiency(
        query=state["query"],
        docs=state["retrieved_docs"]
    )
    return {**state, "context_sufficient": sufficient}

def generate_node(state: AgentState) -> AgentState:
    """Generate answer from retrieved context."""
    answer = generate_with_citations(
        query=state["query"],
        docs=state["retrieved_docs"]
    )
    return {**state, "answer": answer}

def should_retrieve(state: AgentState) -> str:
    if state["context_sufficient"]:
        return "generate"
    if state["iterations"] >= state["max_iterations"]:
        return "generate"  # give up and generate with what we have
    return "retrieve"

# Build the graph
workflow = StateGraph(AgentState)
workflow.add_node("plan",     plan_node)
workflow.add_node("retrieve", retrieve_node)
workflow.add_node("evaluate", evaluate_node)
workflow.add_node("generate", generate_node)

workflow.set_entry_point("plan")
workflow.add_edge("plan",     "retrieve")
workflow.add_edge("retrieve", "evaluate")
workflow.add_conditional_edges("evaluate", should_retrieve,
    {"retrieve": "retrieve", "generate": "generate"})
workflow.add_edge("generate", END)

agent = workflow.compile()

# Run
result = agent.invoke({
    "query": "What were the main causes of the 2023 banking crisis?",
    "max_iterations": 3,
    "retrieved_docs": [],
    "iterations": 0,
})

Listing 17:LangGraph-based agentic RAG

Tool-Augmented RAG

Agentic RAG can combine retrieval with computation tools:

from langchain.agents import create_tool_calling_agent, AgentExecutor
from langchain.tools import tool

@tool
def search_documents(query: str) -> str:
    """Search internal document knowledge base."""
    docs = vectorstore.similarity_search(query, k=5)
    return "\n\n".join(d.page_content for d in docs)

@tool
def query_database(sql: str) -> str:
    """Execute SQL query on the analytics database."""
    return db.run(sql)

@tool
def web_search(query: str) -> str:
    """Search the web for current information."""
    return tavily_client.search(query)

@tool
def execute_python(code: str) -> str:
    """Execute Python code for calculations."""
    return python_repl.run(code)

tools = [search_documents, query_database, web_search, execute_python]
agent = create_tool_calling_agent(llm, tools, prompt)
executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

Listing 18:Tool-augmented RAG with SQL and retrieval

Search-R1: RL-Trained Agentic RAG

The agentic RAG approaches above rely on prompt-engineered orchestration — the agent’s search behavior is controlled by instructions, not learned through training. Search-R1[170] takes a fundamentally different approach: it trains the LLM via reinforcement learning to learn when, what, and how many times to search as part of its reasoning process.

Search-R1 extends the DeepSeek-R1 [69] reasoning framework by treating search engine queries as actions within the RL training loop. During chain-of-thought generation, the model can emit special tokens <search>query</search> that trigger real-time retrieval from a search engine. The retrieved results are injected back into the reasoning context, and the model continues generating.

The model generates a reasoning trace interleaved with search actions:

The entire trajectory (reasoning + searches + final answer) is scored by a terminal reward: correctness of the final answer against a ground-truth label.

Search-R1 uses GRPO (Group Relative Policy Optimization):

1. Sample $N$ trajectories per question, each potentially containing 0–5 search calls
2. Execute searches in real-time — the environment returns actual search engine results
3. Score terminal answer correctness (exact match or F1 against ground truth)
4. Compute group-relative advantage: ${\widehat{A}}_i=(R_i-{\mu }_G)/{\sigma }_G$
5. Update policy with GRPO clipped objective — reinforcing trajectories that searched effectively

The model learns to:

• Search when uncertain — avoid unnecessary searches for knowledge it already has
• Formulate effective queries — learn query phrasing that returns relevant results
• Search multiple times — iteratively refine queries based on initial results
• Integrate retrieved context — use search results to support or correct its reasoning

On open-domain QA benchmarks (NQ [195], TriviaQA [173], HotpotQA [406]), Search-R1 with a 7B model outperforms:

• Standard RAG (single retrieval) by 15–20% accuracy
• Prompted agentic RAG (ReAct-style) by 8–12% accuracy
• Approaches the performance of much larger models (70B) with standard RAG

The key insight: learning when and how to search is more valuable than having a larger model that knows more. A small model that searches well beats a large model that doesn’t search.

Evaluation

Evaluating a RAG system is harder than evaluating retrieval or generation in isolation, because errors can originate at any stage of the pipeline—and they compound. A perfect generator cannot compensate for irrelevant retrievals, and a perfect retriever is wasted if the generator hallucinates or ignores the context.

Effective RAG evaluation therefore operates at three levels:

1. Retrieval quality: Did the retriever surface the right passages? (Recall, Precision, MRR, NDCG)
2. Generation quality: Is the answer correct, faithful to the retrieved context, and complete? (Correctness, Faithfulness, Answer Relevance)
3. End-to-end quality: Does the full system satisfy the user? (Human preference, task success rate, latency-adjusted utility)

A common failure mode is optimizing only one level—for example, maximizing Recall@$K$ with large $K$ fills the context with marginally relevant passages that actually degrade generation quality. The metrics below cover both retrieval and generation, enabling practitioners to diagnose which stage is the bottleneck.

Retrieval Metrics

Let ${ℛ}_k$ be the set of retrieved documents at rank $k$, and ${ℛ}^{\ast }$ be the set of relevant documents.

where ${\text{rank}}_i$ is the rank of the first relevant document for query $i$.

where ${\text{rel}}_i\in \{0,1,2,\mathrm{\dots }\}$ is the graded relevance of the $i$-th result and IDCG is the ideal (perfect) DCG.

Generation Metrics

Measures whether the generated answer is grounded in the retrieved context—i.e., every claim in the answer can be attributed to a retrieved document. Evaluated by an LLM judge:

Measures whether the answer addresses the question. Computed by generating questions from the answer and measuring similarity to the original query:

where ${\widehat{q}}_i$ are questions generated from the answer.

RAGAs Framework

RAGAs (Retrieval Augmented Generation Assessment) [85] provides a reference-free evaluation framework using LLM judges:

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
    answer_correctness,
)
from datasets import Dataset

eval_dataset = Dataset.from_dict({
    "question":  questions,
    "answer":    generated_answers,
    "contexts":  retrieved_contexts,   # list of lists
    "ground_truth": reference_answers,
})

results = evaluate(
    dataset=eval_dataset,
    metrics=[
        faithfulness,
        answer_relevancy,
        context_precision,
        context_recall,
        answer_correctness,
    ],
)
print(results.to_pandas())

Listing 19:RAGAs evaluation (v0.1 API; v0.2+ uses user_input, response, retrieved_contexts, reference)

Common Failure Modes

Production Considerations

Embedding Model Selection

The embedding model is the single most impactful component choice in a RAG system—it determines the quality ceiling for retrieval. The field has advanced rapidly; Table 表 16.5 summarizes current options across the cost–quality spectrum.

• Domain match: Specialized models (e.g., voyage-code-3 for code, voyage-finance-2 for finance) can outperform general models by 5–15% on domain tasks.
• Context length: Models with 32K token context (Voyage-4, NV-Embed-v2) can embed entire documents without chunking, simplifying the pipeline.
• Matryoshka embeddings: Models supporting flexible output dimensions (256–4096) let you trade quality for storage/latency at serving time without re-encoding.
• Quantization support: int8 or binary quantization at the model level (Cohere, Voyage) reduces index size by 4–32$\times$ with minimal recall loss.
• Multilingual: For non-English or cross-lingual RAG, prefer models explicitly trained multilingual (BGE-M3, Jina-v3, Voyage-4).

Vector Database Comparison

5https://qdrant.tech 6https://milvus.io 7https://github.com/pgvector/pgvector

Latency Optimization

1. Pre-filtering: Use metadata filters (date range, category, source) to reduce the search space before ANN search
2. Approximate NN: Use HNSW or IVF indices instead of exact search; accept $\sim$1% recall loss for $10\times$ speedup
3. Embedding caching: Cache embeddings for frequently repeated queries
4. Async retrieval: Retrieve from multiple sources in parallel
5. Streaming generation: Stream LLM output while retrieval completes
6. Quantization: Use int8 or binary quantization for embeddings to reduce memory and increase throughput

Techniques (3) and (4) above compose naturally: cache the query embedding, then fan out retrieval requests to multiple backends concurrently. In a multi-source RAG system (Section 16.7 Agentic RAG), the user query may need results from a vector database, a keyword index, and a web API. Sequential retrieval adds latencies; parallel retrieval pays only the cost of the slowest source. Listing  demonstrates this pattern using Python’s asyncio—the lru_cache decorator ensures repeated queries skip the embedding model entirely, while asyncio.gather dispatches all source queries simultaneously.

import asyncio
from functools import lru_cache

@lru_cache(maxsize=1024)
def get_cached_embedding(text: str) -> list[float]:
    return embedding_model.embed_query(text)

async def parallel_retrieve(
    query: str,
    sources: list[str],
    k: int = 5
) -> list[dict]:
    """Retrieve from multiple sources in parallel."""
    tasks = [
        asyncio.create_task(retrieve_from_source_async(query, src, k))
        for src in sources
    ]
    results = await asyncio.gather(*tasks, return_exceptions=True)
    # Flatten and deduplicate
    all_docs = []
    for r in results:
        if not isinstance(r, Exception):
            all_docs.extend(r)
    return deduplicate_by_content(all_docs)

Listing 20:Async parallel retrieval for low latency

Incremental Indexing and Versioning

In production, the document corpus is never static—policies get revised, new reports land daily, deprecated content must be removed. A full re-index (re-chunk, re-embed, re-upload) is expensive and causes downtime. Incremental indexing solves this by applying changes at the document level.

• Upsert: When a document is created or updated, delete all existing chunks for that doc_id, re-chunk the new content, embed, and insert. This guarantees no stale fragments linger.
• Delete/Expire: Remove chunks by document ID (explicit deletion) or by TTL (automatic garbage collection for time-sensitive sources like news or market data).
• Version tracking: Store a version and indexed_at timestamp in chunk metadata. This enables rollback (restore previous version from source) and auditability (“which version did the model see?”).

• Embedding model drift: If you upgrade the embedding model, old and new vectors are incompatible. Solutions: (a) maintain separate indices per model version and migrate in the background, or (b) use Matryoshka-compatible models where dimension truncation preserves compatibility.
• Chunk boundary shifts: Changing the chunking strategy invalidates all existing chunks. Version metadata lets you identify and selectively re-index affected documents.
• Eventual consistency: In distributed vector databases, newly upserted vectors may not be immediately searchable. Design your pipeline to tolerate a brief indexing lag (typically seconds to minutes).

Listing 21 shows a minimal RAGIndexManager class that encapsulates upsert and expiration logic, suitable for wrapping any vector store with metadata filtering support.

class RAGIndexManager:
    def __init__(self, vectorstore, metadata_store, chunker, embedder):
        self.vs = vectorstore
        self.meta = metadata_store
        self.chunker = chunker
        self.embedder = embedder

    def upsert_document(self, doc_id: str, content: str,
                        metadata: dict) -> None:
        """Add or update a document, replacing old chunks."""
        # Delete existing chunks for this document
        self.vs.delete(filter={"doc_id": doc_id})
        # Chunk new version (vectorstore embeds internally)
        chunks = self.chunker.split_text(content)
        self.vs.add_texts(
            texts=chunks,
            metadatas=[{**metadata, "doc_id": doc_id,
                        "version": metadata.get("version", 1),
                        "indexed_at": datetime.utcnow().isoformat()}
                       for _ in chunks],
        )

    def expire_old_documents(self, ttl_days: int = 365) -> int:
        """Remove documents older than TTL."""
        cutoff = (datetime.utcnow() - timedelta(days=ttl_days)).isoformat()
        return self.vs.delete(filter={"indexed_at": {"$lt": cutoff}})

Listing 21:Incremental index updates with versioning

RAG + Fine-Tuning Synergy

When to Combine RAG with Fine-Tuning

Fine-tuning and RAG address complementary weaknesses:

• Fine-tuning alone: model learns style and format but may hallucinate facts
• RAG alone: model has access to facts but may not know how to use them optimally
• Combined: fine-tune the model to use retrieved context well—cite sources, acknowledge uncertainty, and ignore irrelevant context

RAFT: Retrieval-Augmented Fine-Tuning

RAFT [425] trains models to answer questions given a mix of relevant and distractor documents, teaching the model to identify and use only the relevant context:

1. For each training example $(q,a,d^{\ast })$, sample $k-1$ distractor documents $\{d_i^{-}\}$
2. Fine-tune on: [q, $d^{\ast }$, $d_1^{-}$, …, $d_{k-1}^{-}$]$\to$[chain-of-thought + a]
3. The chain-of-thought explicitly quotes from $d^{\ast }$, teaching the model to ground answers

Joint Retriever-Generator Training

For maximum performance, the retriever and generator can be trained jointly. The REALM [128] and RAG [209] papers propose end-to-end training where gradients flow through the retrieval step:

The retriever parameters $\varphi$ are updated using the REINFORCE estimator or by treating $P_{\varphi }(d\mid q)$ as a differentiable attention over documents.

Comprehensive RAG Approach Comparison