Chapter 12 LLM Agentic Training

Motivation: From Chatbots to Autonomous Agents

Modern LLMs are increasingly deployed not just as conversational assistants but as autonomous agents that interact with external tools, APIs, databases, and environments over multiple steps. This shift—from single-turn chatbots to multi-step agents—introduces fundamentally new RL challenges that require rethinking how we train, evaluate, and deploy language models.

The key differences that demand new RL approaches:

• Multi-step reasoning: Agents must plan across 10–100+ tool calls, not just generate a single response.
• External environment feedback: Rewards come from real-world execution (test suites pass, web pages load, code compiles) — not just human preference scores.
• Structured actions: Actions are not just tokens but structured outputs (JSON tool calls, API payloads, code blocks).
• Long horizons with sparse rewards: Success/failure may only be determined after many intermediate steps.

Trajectory Buffers for LLM Agents

In the context of LLM agents, traditional RL replay buffers undergo a structural transformation. Instead of storing low-dimensional numerical tensors, agentic buffers — often called Trajectory Buffers, Experience Pools, or Memory Banks — manage complex textual histories, tool execution outputs, and explicit reasoning steps.

Mathematical Structure of an LLM Agent Buffer

In classic RL, a replay buffer stores a flat tuple $(s,a,r,s^{\prime })$. For an LLM agent, this expands into high-dimensional tokenized text structures:

• ${𝒮}_t$: The complete context state — system prompt, user objective, conversation history, and current environmental variables (e.g., HTML source code, directory structures, database schemas).
• ${𝒜}_t$: The agent’s generated output, typically composed of a Chain-of-Thought (CoT) reasoning string followed by a structured tool call:$${𝒜}_t=\{{\text{text}}_{\text{reasoning}},{\text{json}}_{\text{tool\_call}}\}$$(12.2)
• ${ℛ}_t$: Evaluation signals derived from external execution environments (unit test passes, compiler flags, API response codes) or verified by an LLM-as-a-judge system.
• ${𝒮}_{t+1}$: The updated context window, which appends tool output text or error logs directly into the conversation history.

Operational Paradigms

LLM agents leverage specialized trajectory buffers through three primary optimization methodologies:

A. Self-Correction and Thought Refinement

Two representative methods in this category are STaR [419] and Reflexion [326]. When an agent fails a multi-step execution trace, the sub-optimal sequence is saved to the buffer. The framework later samples this trajectory and prompts the LLM to generate an explicit textual critique of its past performance:

Once a corrected trajectory achieves a positive reward, it is moved to an optimal experience pool used to update the network weights via fine-tuning (SFT on successful trajectories) or RL (GRPO [323] with binary pass/fail rewards).

B. Off-Policy Exploration

This paradigm, exemplified by ReAct [409] and related tool-use frameworks, involves extensive autonomous exploration. During autonomous exploration (web navigation, database querying, code generation), agents log thousands of exploratory execution paths. The trajectory buffer acts as a filter:

• Success filtering: Only trajectories achieving the goal are kept for training.
• Efficiency ranking: Among successful traces, prefer the shortest/most efficient tool-use paths.
• Diversity sampling: Maintain a diverse set of solution strategies to prevent mode collapse.

The optimization algorithm (typically GRPO [323] or filtered SFT) computes losses exclusively over efficient, successful trajectories while discarding meandering runs.

C. Non-Parametric In-Context Learning (RAG over Experiences)

Instead of modifying neural network weights, the trajectory buffer can function as a vector database. Given a new user goal ${𝒢}_{\text{new}}$, the system retrieves the most relevant past experiences:

The top-$k$ similar successful historical runs are injected directly into the prompt context as few-shot demonstrations. This approach:

• Requires zero training — pure retrieval-augmented generation
• Adapts instantly to new tasks if similar experiences exist in the buffer
• Scales with buffer size (more experiences = better coverage)
• Complements parametric learning (use retrieval for rare cases, weights for common patterns)

Paradigm Comparison

Major Techniques in Agentic RL

STaR: Self-Taught Reasoner (Detailed)

STaR [419] is an iterative self-improvement method that bootstraps reasoning capabilities without external reward models. The core insight: if the model can occasionally solve a problem correctly, it can learn from its own successes.

Algorithm:

1. Generate: For each problem $x_i$ in dataset $𝒟$, sample a reasoning trace $z_i\sim {\pi }_{\theta }(\cdot |x_i)$ followed by an answer ${\widehat{y}}_i$.
2. Filter: Keep only traces where ${\widehat{y}}_i=y_i^{\ast }$ (correct answer). Define success set ${𝒟}_{\text{pass}}=\{(x_i,z_i,y_i^{\ast }):{\widehat{y}}_i=y_i^{\ast }\}$.
3. Rationalization (key innovation): For problems where the model failed, generate a “rationalization” — a trace conditioned on the correct answer: $z_i^{\text{rat}}\sim {\pi }_{\theta }(\cdot |x_i,y_i^{\ast })$. This teaches the model to reason backward from solutions.
4. Fine-tune: Update $\theta$ via SFT on ${𝒟}_{\text{pass}}\cup {𝒟}_{\text{rationalized}}$.
5. Iterate: Repeat from step 1 with the improved model.

Convergence dynamics: Each iteration $k$ increases the model’s solve rate $p_k$. If $p_0=0.3$ (solves 30% of problems), after rationalization + SFT, $p_1\approx 0.5$. Typically converges in 3–5 iterations to $p\approx 0.7$–$0.9$.

Reflexion: Verbal Reinforcement Learning (Detailed)

Reflexion [326] introduces a radical paradigm: RL without weight updates. Instead of gradient-based learning, the agent improves through natural-language self-critique stored in an episodic memory.

Full Architecture:

1. Actor: The LLM agent $\pi$ that executes actions in the environment.
2. Evaluator: A binary signal (task success/failure) or a scalar heuristic (e.g., number of test cases passed).
3. Self-Reflection Generator: Given the failed trajectory ${\tau }_{\text{fail}}$ and environment feedback, generates a natural-language reflection $r_{\text{text}}$:$$r_{\text{text}}={\text{LLM}}_{\text{reflect}}({\tau }_{\text{fail}},\text{feedback},\text{task})$$(12.6)
4. Episodic Memory: A sliding window buffer $ℳ=[r_1,r_2,\mathrm{\dots },r_m]$ of past reflections (typically $m\le 3$ to fit in context).
5. Retry Loop: On the next attempt, reflections are injected into the prompt:$$a_{t+1}\sim \pi (\cdot |\text{task},ℳ,\text{current\_state})$$(12.7)

Example reflection: “In my previous attempt, I called the search API before validating the input format, which caused a 400 error. Next time, I should validate the JSON schema first, then make the API call.”

Strengths and Limitations:

StrengthsLimitationsZero gradient computation; works with frozen API models (GPT-4)Limited to context window; can’t accumulate infinite knowledgeFast iteration (seconds per retry vs. hours for RL training)No generalization to unseen tasks (memory is task-specific)Interpretable: human-readable self-correctionsRelies on the model’s existing ability to identify errorsComposes with any base agent architectureDegrades when base model is too weak to generate useful critiques

ReAct: Reasoning + Acting (Detailed)

ReAct [409] establishes the dominant prompting paradigm for tool-using agents by interleaving explicit reasoning steps with environment actions in a single generation stream.

Generation format:

Formal definition: A ReAct trajectory is $\tau =(t_1,a_1,o_1,t_2,a_2,o_2,\mathrm{\dots })$ where:

• $t_i$: Thought (internal reasoning, not executed)
• $a_i$: Action (tool call, executed in environment)
• $o_i$: Observation (environment response, appended to context)

Why it works: Thoughts create an “inner monologue” that helps the model plan before acting, reducing impulsive tool calls. The explicit reasoning trace also makes the agent’s decision process auditable and debuggable.

Training ReAct agents with RL:

• Action-level rewards: Only actions receive reward signals (thoughts are auxiliary).
• Thought quality: Implicitly optimized — better thoughts $\to$ better actions $\to$ higher rewards.
• Format enforcement: Include format penalties in the reward for malformed actions (missing JSON, hallucinated tools).
• RL objective: $r(\tau )=r_{\text{task}}-{\lambda }_{\text{format}}\cdot \text{format\_violations}-{\lambda }_{\text{length}}\cdot \text{num\_steps}$

LATS: Language Agent Tree Search (Detailed)

LATS [438] applies Monte Carlo Tree Search (MCTS) to LLM agent action selection, trading inference compute for significantly better trajectories.

Algorithm (adapted for LLM agents):

1. Selection: Starting from root (initial state), traverse the tree using UCB1:$$\text{UCB}(s,a)=\overline{Q}(s,a)+c\sqrt{\frac{\ln N(s)}{N(s,a)}}$$(12.8)where $\overline{Q}$ = average reward of subtree, $N$ = visit counts, $c$ = exploration constant.
2. Expansion: At a leaf node, generate $k$ candidate actions via LLM sampling (temperature $\phantom{}>0$): $\{a_1,\mathrm{\dots },a_k\}\sim {\pi }_{\theta }(\cdot |s_{\text{leaf}})$
3. Simulation: For each candidate, execute the action in the environment and continue with a fast rollout policy (greedy decoding) until terminal state or depth limit.
4. Backpropagation: Propagate the terminal reward up through all ancestor nodes, updating $\overline{Q}$ and $N$ counts.
5. Repeat: Run steps 1–4 for a fixed computation budget (e.g., 50–200 iterations).
6. Action selection: Choose the most-visited child of the root.

LLM-specific adaptations:

• Value function: Use a separate LLM call to estimate state value: “On a scale of 0–1, how likely is this state to lead to task success?”
• Reflection-based pruning: When a branch fails, generate a reflection and prune similar branches.
• Caching: Store LLM outputs at each node to avoid redundant generation during backtracking.
• Depth budget: Limit tree depth to 10–20 steps (agents rarely need more).

Performance: On WebShop (web navigation), LATS achieves 75% success vs. ReAct’s 40%. On HumanEval (code), pass@1 improves from 68% $\to$ 94% with tree search. The cost: 10–50$\times$ more inference FLOPs per task.

AgentQ: DPO on Agent Trajectories (Detailed)

AgentQ [294] bridges offline preference learning (DPO) with online agent execution by automatically generating preference pairs from trajectory outcomes.

Pipeline:

1. Rollout: Execute $N$ trajectories per task using the current policy ${\pi }_{\theta }$.
2. Evaluate: Score each trajectory with execution-based reward (binary pass/fail or scalar metric).
3. Pair construction: For each task, construct preference pairs:$$({\tau }_w,{\tau }_l)\text{ where }r({\tau }_w)>r({\tau }_l)$$(12.9)Among trajectories for the same task, the one with highest reward = chosen; lowest = rejected.
4. DPO update: Apply standard DPO loss over trajectory-level log-probabilities:$${ℒ}_{\text{AgentQ}}=-\log \sigma \left(\beta \left[\log \frac{{\pi }_{\theta }({\tau }_w)}{{\pi }_{\text{ref}}({\tau }_w)}-\log \frac{{\pi }_{\theta }({\tau }_l)}{{\pi }_{\text{ref}}({\tau }_l)}\right]\right)$$(12.10)
5. Iterate: Updated ${\pi }_{\theta }$ generates new (better) trajectories in the next round.

Key design choices:

• MCTS-guided exploration: Use LATS during rollout phase to generate diverse, high-quality trajectories (better training data).
• Step-level DPO: Instead of comparing full trajectories, compare at the action level — given the same prefix, which next action leads to success?
• Self-play improvement: Each DPO iteration produces a better policy that generates better trajectories that produce better training pairs — a virtuous cycle.

Results: On WebShop, AgentQ achieves absolute 50% $\to$ 82% success rate improvement over the base policy in 3 DPO iterations.

Voyager: Lifelong Learning via Skill Libraries (Detailed)

Voyager [362] introduces compositional skill accumulation — the agent builds a growing library of reusable code functions that serve as high-level actions.

Architecture:

1. Automatic Curriculum: An LLM proposes progressively harder tasks based on the agent’s current skill inventory: “You can now mine wood and craft planks. Next challenge: build a crafting table.”
2. Skill Generation: For each task, the agent writes a JavaScript function (executable code) that solves it:$${\text{skill}}_i=\text{LLM}({\text{task}}_i,\text{environment\_docs},\text{error\_feedback})$$(12.11)
3. Verification: Execute the code in the environment. If it succeeds, add to the skill library. If not, iterate with error feedback (up to 5 retries).
4. Skill Library (vector DB): Each verified skill stored with:•Function signature + docstring (for retrieval)•Embedding of the task description (for semantic search)•Dependencies (which other skills it calls)
5. Retrieval + Composition: For new tasks, retrieve the top-$k$ most relevant skills and compose them:$$\text{solution}=\text{LLM}(\text{new\_task},\text{retrieve}(\text{skill\_library},k=5))$$(12.12)

Key insight: Skills are compositional — complex behaviors emerge from combining simple verified functions. The agent never forgets (library is persistent) and improves monotonically (only verified skills are added).

RLEF: RL from Execution Feedback (Detailed)

RLEF [201] applies online RL with deterministic execution-based rewards to code generation agents, establishing the simplest effective paradigm for agentic training.

Training loop:

1. Sample task: Draw a coding problem with test cases $(x,\text{tests})$ from the training set.
2. Generate: The agent produces a solution trajectory (reading files, writing code, running tests) using the current policy ${\pi }_{\theta }$.
3. Execute: Run the test suite in a sandboxed environment. Reward:$$r=\frac{\text{\# tests passed}}{\text{\# total tests}}\in [0,1]$$(12.13)
4. Update: Apply GRPO/PPO using $r$ as the reward signal.
5. Repeat: Thousands of iterations with fresh tasks.

Why execution feedback is ideal for RL:

• Zero noise: Unlike human preferences, test results are deterministic. Same code $\to$ same reward every time. This eliminates reward noise that destabilizes RL training.
• Infinite scale: Can generate unlimited tasks programmatically (random algorithms, API integration tests, data transformations).
• No reward hacking: Unlike learned reward models, a test suite can’t be “fooled” (assuming tests are well-written). The agent must actually solve the problem.
• Dense signal: Partial test passage ($r=0.6$) provides richer gradient than binary pass/fail.

OpenHands / SWE-Agent: GRPO for Software Engineering

OpenHands [366] and SWE-Agent [404] apply GRPO to train agents that autonomously resolve GitHub issues — reading code, writing patches, and running test suites.

Training specifics:

• Environment: Docker container with full repo, test suite, and developer tools (git, grep, lint).
• Action space: Bash commands, file edits, git operations, test execution.
• Trajectory length: 15–50 actions typical for resolving a GitHub issue.
• Reward: Binary — does the generated patch pass the issue’s regression tests?
• Group size: $N=8$–$16$ trajectories per issue for GRPO normalization.
• Curriculum: Start with issues labeled “good first issue”, progress to complex multi-file refactors.

State-of-the-art results: SWE-bench Verified: 30% $\to$ 55% resolve rate after RL training (vs. SFT-only baseline).

Use Case: Agentic RL for a Productivity Co-pilot

This section provides a complete blueprint for applying agentic RL techniques to train an LLM-based co-pilot that operates across a productivity application suite (documents, spreadsheets, presentations, email, messaging, cloud storage).

Architecture Overview

Formal MDP Definition for a Productivity Co-pilot

The productivity co-pilot environment is formalized as a Partially Observable Markov Decision Process (POMDP):

• $𝒮$: State space — Full workspace environment state: document contents, email threads, calendar events, file system, user permissions. Not fully observable: agent sees only what API queries return.
• $𝒜$: Action space — Structured API calls (see below). Each action is a JSON object specifying the target app, operation, and parameters.
• $𝒯$: Transition function — Deterministic for most operations (write to document $\to$ document updated), but stochastic for network-dependent actions (email delivery time, Teams availability).
• $ℛ$: Reward function — Multi-component (see Reward Design section).
• $\mathrm{\Omega }$: Observation space — API responses, rendered document views, error messages.
• $𝒪$: Observation function — Maps state to observation (API response formatting, truncation for context window limits).
• $\gamma =0.99$: Discount factor (long horizons, 10–50 steps typical).

Action Space Design

The action space must be structured, type-safe, and composable:

Action taxonomy by application:

AppComplexityKey ActionsOutlookMediumsearch, read, draft, send, move, flag, create_ruleExcelHighread_range, write_range, insert_formula, create_chart, pivot_table, run_macroWordMediumread_paragraphs, insert_text, format_section, find_replace, insert_tablePowerPointMediumadd_slide, insert_shape, set_text, set_layout, add_image, apply_themeTeamsLowsend_message, create_meeting, search_chat, add_members, post_to_channelSharePointMediumlist_files, upload, download, search, create_page, set_permissions

State Representation

The agent’s observation (context window) at each step:

Context budget management (critical for 128K window):

• System prompt: 2K tokens (capabilities, safety rules, output format)
• User intent + conversation: 4K tokens
• Tool history (sliding window): Last 8–12 actions + observations, summarizing older ones. Total: 80K tokens max.
• Current observation: Up to 32K tokens (large spreadsheets, email threads)
• Reserve: 10K tokens for agent’s reasoning + next action generation

State compression strategies:

• Selective inclusion: Only include API responses relevant to the current sub-goal (use an auxiliary “relevance scorer”).
• Structured summaries: Represent large spreadsheets as schema + sample rows rather than full data.
• Hierarchical memory: Store full trajectory externally; inject compressed summaries into context.

Reward Design: Multi-Objective Signal

The reward function for a productivity co-pilot must balance multiple objectives:

Intermediate rewards (dense signal):

• Successful API call (200 response): +0.05
• Correct information retrieval (verified by downstream use): +0.10
• Recovers from error gracefully (retries with corrected params): +0.08
• API error (4xx/5xx): –0.03
• Repeated identical action (loop detection): –0.10
• Asks clarifying question when intent is genuinely ambiguous: +0.05

Training Pipeline: End-to-End

Simulation Environment Architecture

Task Curriculum Design

Training effectiveness depends critically on task difficulty progression:

Curriculum strategy:

• Start with 80% L1–L2 tasks, 20% L3 in early training.
• Advance to next level when success rate exceeds 70% on current level.
• Always maintain 10–20% of easier tasks to prevent catastrophic forgetting.
• Final mix (after convergence): 10% L1, 15% L2, 25% L3, 30% L4, 20% L5.

Safety and Guardrails

Credit Assignment in Multi-App Workflows

The key challenge: in a 20-step cross-app workflow, which steps contributed to success or failure?

Approach: Hierarchical Reward Decomposition

1. Sub-goal detection: Decompose the user’s instruction into verifiable sub-goals:•“Find Q4 budget emails” $\to$ Sub-goal 1 (verified: relevant emails retrieved)•“Extract numbers” $\to$ Sub-goal 2 (verified: correct values parsed)•“Create Excel sheet” $\to$ Sub-goal 3 (verified: sheet exists with correct data)
2. Sub-goal rewards: Assign intermediate rewards at each sub-goal completion ($r=+0.2$ each).
3. Trajectory slicing: If the final task fails, identify which sub-goal failed first. Apply negative reward only to the actions within that sub-goal’s span.
4. Counterfactual estimation: “Would the task have succeeded if this specific action were different?” — use the value function to estimate.

Scaling and Infrastructure

Compute requirements (estimated for 70B parameter model):

ComponentResourcesNotesPolicy model (70B)8$\times$ A100 80GB (TP=8)BF16, generates trajectoriesReference model (70B)8$\times$ A100 80GB (TP=8)Frozen, for KL computationEnvironment workers128 CPU workersEach runs sandbox instanceReward model / Judge4$\times$ A100 (if LLM judge)Or zero if using execution-based rewardsTraining (GRPO updates)16$\times$ A100 (FSDP)Gradient accumulation over trajectory batchesTotal40 A100 GPUs + 128 CPUs5,000 GPU-hours for full training run

Throughput optimization:

• Async rollouts: Decouple trajectory generation from gradient updates. Generate continuously while training on previous batch.
• Batched environment: Run 128 sandbox environments in parallel, each processing different tasks.
• KV-cache sharing: For the $N=8$ trajectories per task, they share the same prompt prefix — use prefix caching to avoid redundant computation.
• Selective backprop: Only compute gradients over action tokens (not observations/system prompt). Reduces backward pass FLOPS by 40–60%.

Evaluation Framework

Benchmark suite (proposed):

• ProdBench-Easy (200 tasks): Single-app, 1–3 steps. Baseline establishment.
• ProdBench-Hard (200 tasks): Cross-app workflows, 10–30 steps. End-to-end capability.
• ProdBench-Safety (100 tasks): Adversarial prompts attempting to trigger unsafe actions. Must maintain violation rate.
• ProdBench-Robustness (100 tasks): Tasks with ambiguous instructions, API errors injected, missing permissions. Tests graceful degradation.

Lessons from Production Deployments

Complete Training Recipe

Use Case: Building a Research Agent from Scratch

This use case demonstrates how to build a fully autonomous research agent — an LLM that can formulate hypotheses, search literature, analyze data, write code, run experiments, and produce a final report — using the techniques discussed throughout this paper.

Problem Definition

MDP Formulation

Action Space

Architecture: Model and Infrastructure Choices

Reward Design

Training Pipeline

1. Phase 1 — SFT Warmup (500 steps):•Collect 200 expert research trajectories (human researchers using the tools)•SFT on successful trajectories with completion-only masking (mask tool outputs)•This teaches the agent tool-use syntax and basic research workflow
2. Phase 2 — GRPO Training (3000 steps):•Prompt pool: 500 research questions across 10 domains (ML, NLP, CV, systems, etc.)•Per question: generate $N=4$ complete research trajectories•Score each trajectory with multi-component reward•GRPO advantage: ${\widehat{A}}_i=(R_i-{\mu }_G)/{\sigma }_G$•Update policy with clipped objective (clip $ϵ=0.2$, KL $\beta =0.05$)•Curriculum: start with simple “summarize findings on X” tasks, progress to “design and run experiment on X”
3. Phase 3 — Rejection Sampling Fine-Tuning (200 steps):•Generate 16 trajectories per hard question, keep top-2 by reward•SFT on these high-quality trajectories•Stabilizes performance on the hardest research tasks

Example Trajectory: Full MDP Trace

To illustrate how all the MDP components work together in practice, we trace a complete research episode from question to submission, annotating each step with its formal MDP elements.

Key Design Decisions and Tradeoffs

Evaluation

Lessons and Failure Modes

State-of-the-Art RL for LLM Agents

RL techniques for LLM agents focus on on-policy policy gradients combined with fine-grained credit assignment. Because agents execute complex multi-turn trajectories involving tool interactions, API queries, and code execution, standard single-turn alignment algorithms must be heavily modified.

Dominant Baseline: GRPO for Agents

Popularized by DeepSeek-R1 [69], GRPO[323] is rapidly becoming the standard for agentic training. It samples a group of $N$ complete trajectories per task, eliminating the memory-intensive critic network:

For a task prompt $q$, GRPO samples $N$ agentic trajectories $\{o_1,o_2,\mathrm{\dots },o_N\}$ from ${\pi }_{{\theta }_{\text{old}}}$. The advantage for each trajectory is computed by normalizing its reward relative to the group:

The GRPO objective with KL regularization:

PPO for Interactive Agents

PPO[319] remains valuable for agents operating in highly stochastic environments where step-level value estimation helps. The critic provides per-step advantage signals, enabling finer credit assignment when tool outputs are unpredictable:

• Step-level advantage estimation via GAE handles variable-length tool outputs
• Value head learns to predict “how likely is this trajectory to succeed from here”
• More stable when external tools return catastrophic errors that spike reward variance
• Trade-off: requires $2\times$ memory (critic) but provides denser learning signals

Fine-Grained Turn-Level Credit Assignment

The core challenge in agentic RL is the sparse reward problem. If an agent executes 20 tool actions and finally fails a unit test, a terminal reward of $0$ punishes all 20 actions equally. Modern solutions:

Alternative Paradigms

• Iterative STaR (Self-Taught Reasoner)[419]: Rather than continuous RL, use iterative offline loops. Generate trajectories $\to$ filter failures $\to$ SFT on successes $\to$ repeat. Simple to scale, avoids RL instability. Each iteration bootstraps reasoning ability.
• Reinforcement World Model Learning (RWML)[414]: To combat reward hacking, train agents to predict the semantic consequence of their actions. The agent receives an auxiliary reward for accurately predicting how environment state will change (e.g., predicting database table changes before executing SQL). This forces genuine understanding over superficial reward-gaming.
• LATS (Language Agent Tree Search)[438]: Apply Monte Carlo Tree Search over agent action sequences. At each step, expand multiple candidate actions, simulate their outcomes, and backpropagate rewards through the tree. Combines RL value estimation with search-time compute scaling.

Core Methodology Comparison

Interactive RL Environments for Agent Training

The dominant paradigm in early agentic RL relied on static datasets: curated trajectories, offline preference pairs, or fixed benchmark suites. While tractable, this approach has a fundamental limitation—agents trained on static data cannot discover novel strategies through environment interaction, and reward signals are limited to what was anticipated at dataset creation time. The field has shifted decisively toward interactive RL environments: live simulators where agents execute actions, receive feedback, and update policies based on real outcomes.

NeMo Gym (NVIDIA)

NeMo Gym[271] is NVIDIA’s framework for interactive RL environments designed specifically for LLM agent training. Its architecture cleanly separates the agent (the LLM policy) from the environment (the task simulator), enabling independent scaling of each component.

Key design principles:

• Multi-turn rollouts: Environments support extended interaction sequences, not just single-step queries. The agent can call tools, receive results, and continue reasoning across many turns.
• Tool-calling verification: Rewards are computed by executing the agent’s tool calls and verifying outcomes against ground truth—providing dense, reliable signal.
• Decoupled agent/environment: The environment runs as a separate service, allowing heterogeneous compute allocation (e.g., GPU for the agent, CPU for environment simulation).

Nemotron 3 Super was trained using NeMo Gym with 21 environment configurations spanning math, code, tool use, and multi-turn dialogue, generating 1.2M rollouts. The diversity of environments is critical: agents trained on a single environment type tend to overfit its reward structure.

RLFactory

RLFactory[369] is a plug-and-play framework for multi-round tool-use RL, designed to minimize the engineering overhead of connecting LLM policies to diverse tool environments.

MOSAIC: Safety in Agentic Training

As agentic RL environments become more capable, safety becomes a first-class concern. MOSAIC[50] (March 2026) addresses the problem of training agents that are both capable and safe in interactive environments.

The core MOSAIC framework introduces a plan $\to$ check $\to$ act or refuse loop:

1. Plan: The agent generates a candidate action sequence.
2. Check: A safety verifier evaluates the plan against a constraint set (harm categories, policy rules).
3. Act or Refuse: If the plan passes, execute; otherwise, generate a refusal with explanation.

Safety is trained via trajectory-level preference learning: rather than labeling individual actions as safe/unsafe, MOSAIC labels entire trajectories, capturing the cumulative safety impact of multi-step plans. This is critical because individual actions that appear benign can compose into harmful sequences.

xAI’s Grok 4.5 (July 2026) [389] introduced a novel data source: anonymized developer-session traces from Cursor—the actual keystrokes, navigation patterns, and editing sequences of engineers working in real codebases. Mid-trained on these traces, Grok 4.5 achieved top rankings on agentic tool-use benchmarks. This signals a “compute + traces” paradigm: once compute is scaled, the scarce input becomes recordings of target behavior—analogous to how AlphaGo was bootstrapped from human games before transitioning to self-play. The approach raises unresolved data-rights questions (whose code, under what consent?) but demonstrates that behavioral demonstrations from expert users remain a powerful training signal for agentic capabilities.