Chapter 29 Quick Reference

This chapter consolidates key equations, architecture specifications, API references, and failure mode diagnostics for rapid lookup during development and debugging.

Core RL & Alignment Equations

Transformer & Architecture Formulas

Decoding Methods

MethodFormula / RuleKey ParamGreedy—Beam searchKeep top-$B$ partial sequences by joint probability$B=4$–$8$Temperature$P^{\prime }(v)=\text{softmax}({\text{logit}}_v/T)$$T\in [0.1,1.5]$Top-$k$Zero out all but top-$k$ logits, renormalize$k=40$–$100$Top-$p$ (nucleus)Keep smallest set $V^{\prime }$ s.t. ${\sum }_{v\in V^{\prime }}P(v)\ge p$$p=0.9$–$0.95$Min-$p$Keep tokens with $P(v)\ge p_{\text{min}}\cdot P(v_{\text{max}})$$p_{\text{min}}=0.05$–$0.1$Repetition penalty${\text{logit}}_v\leftarrow {\text{logit}}_v/\theta$ if $v$ appeared before$\theta =1.1$–$1.3$

Systems & Parallelism

FormulaValue (70B, BF16)DescriptionModel memory$2P$ bytes$140$ GB (weights only)Adam optimizer$2P\times 4$ bytes (m + v)$280$ GBFull training footprint$\phantom{}\sim 8P$ bytes$560$ GB (weights + opt + grad)FSDP memory/GPU$8P/N_{\text{GPUs}}$$70$ GB with 8 GPUsGen arithmetic intensity$2P/2P=1$ FLOP/byteHeavily memory-boundToken rate (gen)HBM_BW $\phantom{}/(2P)$$\sim$14 tok/s (A100, batch=1)TP AllReduce / layer$2\times 2\cdot \frac{T-1}{T}\cdot bsd$ bytes$\sim$188 MB (70B, TP=8)PP bubble fraction$(P-1)/(P+M-1)$$P$=stages, $M$=micro-batchesMFUobserved_toks $\times$ 6$P$ / peak_FLOPSTarget: $\phantom{}>40\%$

GPU Hardware Specs

GPUMemoryBW (HBM)BF16 TFLOPSNVLinkNotesA100-80GB80 GB HBM2e2.0 TB/s312600 GB/sWorkhorse, widely availableH100-80GB80 GB HBM33.35 TB/s989900 GB/sCurrent gen, FP8 supportH200-141GB141 GB HBM3e4.8 TB/s989900 GB/sLarge context / fewer GPUsB200192 GB HBM3e8.0 TB/s22501800 GB/sNext gen (2025)

Hyperparameter Ranges

ParameterTypical RangeDefaultNotes$\beta$ (DPO/KTO)0.05–0.50.1Higher = more conservative$ϵ$ (PPO clip)0.1–0.30.2Higher = more aggressive updates$\gamma$ (GAE discount)0.99–1.01.0Use 1.0 for episodic tasks$\lambda$ (GAE)0.9–0.990.95Lower = more biased, less varianceKL coeff (${\beta }_{\text{KL}}$)0.01–0.20.05Auto-adapt to target KL $\approx$ 5–8LR (RLHF)1e-7 – 5e-65e-7Much lower than pre-trainingLR (SFT)1e-5 – 5e-52e-5Standard fine-tuning rangeLoRA rank $r$8–12816–64Higher $r$ = more capacity, more memoryLoRA alpha $\alpha$$r$ – $2r$$2r$Scaling factor; $\alpha /r$ is the effective scaleTemperature (gen)0.6–1.00.7Lower = less diverse candidatesNum generations $K$4–644–16For GRPO/Online DPO/Best-of-NGrad clip norm0.5–2.01.0Prevents gradient explosion

TRL API Quick Reference

TrainerMethodKey ConfigData FormatSFTTrainerSupervised FTpacking, max_seq_lengthprompt + completionRewardTrainerReward modelcenter_rewards_coefficientprompt + chosen + rejectedPPOTrainerPPOinit_kl_coef, target_kl, cliprangeprompts (online gen)DPOTrainerDPO/IPObeta, loss_type="sigmoid"/"ipo"prompt + chosen + rejectedGRPOTrainerGRPOnum_generations, beta, use_vllmprompts + reward_fnOnlineDPOTrainerOnline DPOnum_generations, reward_model_pathprompts (online gen)KTOTrainerKTOdesirable_weight, undesirable_weightprompt + completion + labelORPOTrainerORPObetaprompt + chosen + rejectedBest-of-N (manual)Best-of-Nn_samplesprompts (inference)

RAG Pipeline Formulas

Agentic Design Patterns

PatternStructureBest ForReActThink $\to$ Act $\to$ Observe $\to$ loopGeneral tool-use agentsPlan-and-ExecutePlan $\to$ Execute steps $\to$ ReviseLong-horizon, structured tasksSupervisorRouter $\to$ specialist agentsMulti-domain, clear subtask boundariesSwarm (handoffs)Agent transfers control + contextCustomer service, escalation flowsHierarchicalTree of delegating agentsComplex decompositionHuman-in-the-loopAgent $\to$ Approval gate $\to$ ContinueHigh-stakes, irreversible actions

Agent Communication Protocols

ProtocolScopeTransportKey ConceptMCPTool integrationstdio / HTTP+SSEServer exposes tools; client discovers & callsA2AAgent-to-agentHTTP + JSON-RPCTasks with lifecycle (submitted$\to$working$\to$done)OpenAI Function CallingTool useAPI payloadJSON schema in tools[] array

Context Window Budget

Rule of thumb for 128K context:

• System prompt: 1–4K tokens (fixed)
• Tool definitions: 2–8K (scales with # tools)
• RAG context: 4–16K (top-$k$ chunks)
• History: grows unbounded $\to$ summarize/truncate
• Reserved output: 2–8K

Common Failure Modes & Fixes

SymptomLikely CauseFixReward up, quality downReward hackingRM ensemble, length penalty, increase $\beta$KL exploding ($>$15)LR too high or mode collapseReduce LR, checkpoint rollbackEntropy collapsePremature convergenceIncrease entropy coeff, raise temperatureTraining loss NaNGradient explosionReduce LR, increase grad clip, check dataNo improvement after 5K stepsBad prompt distributionGoldilocks filter (20–80% pass rate)Benchmark regressionAlignment taxReduce RL budget, use LoRA, mix SFT dataLength increasing monotonicallyLength exploit in RMLength penalty, retrain RM with length controlOOM during generationKV cache overflowReduce batch, increase TP, PagedAttentionAgent loops foreverNo max-iteration guardSet max_iterations, add loop detectionTool call parse failuresInconsistent output formatFew-shot examples, constrained decodingRAG returns irrelevant docsPoor embedding / chunkingReranker, hybrid search, smaller chunksMulti-agent deadlockCircular dependenciesDAG enforcement, timeout per agent

Method Selection Decision Tree

1. Have paired preferences (chosen + rejected)?•Noisy labels $\to$IPO•Memory-constrained, no SFT done yet $\to$ORPO•Clean data, limited compute $\to$DPO•DPO plateaus, want exploration $\to$Online DPO
2. Have only binary feedback (thumbs up/down)?$\to$KTO
3. Have verifiable rewards (math/code)?$\to$GRPO
4. Need maximum quality, any cost?$\to$PPO
5. Want training-free improvement?$\to$Best-of-N

Evaluation Metrics

MetricRangeWhat It MeasuresPerplexity$[1,\mathrm{\infty })$Model’s surprise; lower = better language modelingWin Rate (vs. baseline)$[0,1]$Fraction of outputs preferred by judge/humanBLEU$[0,1]$$n$-gram overlap with reference (precision-focused)ROUGE-L$[0,1]$Longest common subsequence with referencePass@$k$$[0,1]$Probability $\ge$1 of $k$ code samples passes testsMMLU / GPQA$[0,1]$Multi-choice accuracy on knowledge/reasoning benchmarksHumanEval$[0,1]$Functional correctness of generated codeFaithfulness (RAG)$[0,1]$Fraction of claims supported by retrieved contextContext Relevancy$[0,1]$Fraction of retrieved content relevant to queryAnswer Relevancy$[0,1]$Degree to which answer addresses the question

Reasoning & Test-Time Scaling

MethodCompute CostMechanismChain-of-Thought (CoT)1.5–3$\times$ tokens“Think step by step” in promptSelf-Consistency$N\times$ generationSample $N$ CoT paths, majority vote on final answerTree-of-Thought (ToT)$B\times D\times$ generationBFS/DFS over reasoning tree; evaluate branchesBest-of-$N$$N\times$ generationSample $N$, score with RM, pick highestBeam search (on reasoning)$B\times$ generationMaintain top-$B$ partial reasoning chainsBudget forcingVariableAllocate more tokens to harder problems dynamicallyVerification (ORM/PRM)$N\times$ gen + scoringGenerate $N$ solutions, rank by outcome/process RM

Memory System Types

TypeStorageUse CaseWorking memoryContext windowCurrent conversation, immediate tool resultsEpisodic memoryVector storePast interactions, user preferences, session historySemantic memoryKnowledge graph / embeddingsFacts, concepts, domain knowledgeProcedural memorySkill library / codeHow-to procedures, learned workflows

MCP Quick Reference

PrimitiveDirectionSide Effects?PurposeToolsClient $\to$ ServerYesExecute actions (create, modify, delete)ResourcesClient $\to$ ServerNo (read-only)Read data (files, DB records, configs)PromptsClient $\to$ ServerNoReusable templates for common tasksSamplingServer $\to$ ClientNoServer requests LLM generation from client

Transport: stdio (local subprocess) or HTTP+SSE (remote, streamable).

Discovery: Client calls tools/list, resources/list, prompts/list at connection init.

Tool annotations: readOnlyHint, destructiveHint, idempotentHint, openWorldHint.

A2A Protocol Quick Reference

ConceptDescriptionAgent CardJSON at /.well-known/agent.json — name, skills, supported content typesTaskUnit of work: id, status, artifacts. Lifecycle: submitted $\to$ working $\to$ completed/failedMessageCommunication unit within a task (role: user/agent, parts: text/file/data)ArtifactOutput produced by the agent (structured data, files, generated content)Push NotificationsWebhook-based updates for long-running tasks (via tasks/pushNotification/set)

Key endpoints: tasks/send (create/update), tasks/get (poll status), tasks/sendSubscribe (SSE stream).

Agent Framework Comparison

FrameworkOrchestrationMulti-AgentBest ForLangGraphExplicit state graphConditional routingProduction: persistence, HITL, fine controlOpenAI Agents SDKDeclarative handoffsHandoff-basedSimplicity: guardrails, tracing, fast startAutoGen (AG2)Conversation-drivenGroupChatPrototyping: code execution, researchCrewAIRole-based teamsSequential/parallelLow-code: quick demos, simple pipelinesGoogle ADKSession + eventsA2A nativeEnterprise: artifact mgmt, multi-modal

Agentic RL Formulas

Agent Security Checklist

ThreatLayerMitigationPrompt injection (direct)InputInput validation, instruction hierarchy, delimitersPrompt injection (indirect)Tool outputTreat tool output as untrusted; don’t follow instructions in retrieved docsTool misuseExecutionLeast-privilege permissions; destructiveHint gates; sandboxingData exfiltrationOutputOutput filtering; restrict tool access to allowed domainsExcessive autonomyArchitectureMax iterations; cost budgets; human approval gatesConfused deputyMulti-agentVerify task origin; capability-based access control

Agent Evaluation Metrics

MetricFormula / DefinitionTargetTask Success Rate (TSR)Correct completions / total tasks$\phantom{}>85\%$ (production)Steps to completionAvg agent actions per successful taskLower = more efficientCost per taskTotal tokens $\times$ price/tokenBudget-dependentLatency (TTFC)Time from request to first useful outputs for interactiveTool call accuracyCorrect tool selections / total calls$\phantom{}>90\%$Recovery rateSuccessful retries / initial failures$\phantom{}>60\%$Human escalation rateTasks requiring human / total tasks

Key Agentic Benchmarks

BenchmarkDomainMetricSOTA (2025)SWE-bench VerifiedSoftware engineering% resolved issues$\sim$70%WebArenaWeb browsingTask success rate$\sim$40%OSWorldDesktop computer useTask success rate$\sim$25%GAIAGeneral AI assistantExact match accuracy$\sim$75% (L1)Tau-benchTool-use reliabilityPass rate (5 trials)$\sim$65%HumanEval / MBPPCode generationPass@1$\phantom{}>95\%$