Chapter 30 Conclusion and Future Directions

Summary

This guide has traced the full arc from transformer foundations through reinforcement learning for alignment to the construction of autonomous agentic systems. The key themes that emerge across all chapters:

1. Alignment is a systems problem. It is not enough to have a good loss function. Production RLHF requires managing 4+ models, distributing computation across hundreds of GPUs, handling fault tolerance, and monitoring for reward hacking—all simultaneously.
2. There is no single best method. PPO remains the gold standard for maximum quality but demands enormous engineering investment. DPO and its variants offer compelling trade-offs for teams with limited infrastructure. GRPO bridges the gap for verifiable-reward domains. The right choice depends on your data, compute budget, and quality bar.
3. Reasoning emerges from reward. DeepSeek-R1 proved that chain-of-thought, self-verification, and backtracking can emerge from simple binary reward signals and group-relative optimization—without explicit demonstrations of reasoning. Test-time compute scaling means smaller models with more thinking can match larger models.
4. Standards unlock ecosystems. MCP reduces the tool integration problem from $N\times M$ to $N+M$. A2A enables agents built by different teams to collaborate without shared internals. These protocols are to agentic AI what HTTP was to the web—the enabling infrastructure for an open ecosystem.
5. Agents are the natural next step. Once a model is aligned, the frontier shifts from “how good is a single response?” to “can the model solve multi-step problems autonomously?” This requires new training paradigms (agentic RL with environment rewards), new infrastructure (harnesses, tool protocols, memory systems), and new evaluation methods (trajectory-level benchmarks).
6. Evaluation drives everything. Without rigorous evaluation—from reward model validation to agent task success rates, from contamination detection to LLM-as-Judge calibration—progress is unmeasurable and regressions are invisible. The benchmarks you choose shape the systems you build.
7. Simplicity scales. The most reliable production agents use the simplest architecture that meets requirements—prompt chaining and routing before autonomous loops, single agents before multi-agent swarms. Complexity should be earned through demonstrated need.

The Road Ahead: Open Challenges

Learning from Interaction

Current RLHF pipelines [280] treat alignment as a one-time training phase. The future points toward continuous learning from deployment: agents that improve from every user interaction, tool failure, and environment observation—without catastrophic forgetting [186] or reward drift. Key open problems:

• Online learning with non-stationary reward distributions.
• Safe exploration in production [107] (avoiding harmful actions while learning).
• Efficient credit assignment over long agent trajectories (hundreds of tool calls).

Scalable Oversight

As agents become more capable, human oversight becomes the bottleneck. Current approaches (RLHF [280], Constitutional AI [16]) rely on humans evaluating model outputs—but what happens when model outputs exceed human understanding?

• Recursive reward modeling[57]: Use AI to help humans evaluate AI.
• Debate and amplification[157]: Two models argue; a human judges which argument is more compelling.
• Process-based supervision[216]: Reward correct reasoning steps, not just final answers.
• Mechanistic interpretability[273]: Understand what the model is doing internally, not just what it outputs.

World Models and Planning

Current agents are reactive—they observe and respond one step at a time. Future agents will need internal world models[130] that enable lookahead planning:

• Predicting the consequences of actions before executing them.
• Tree search over possible action sequences (à la AlphaGo [331] and MuZero [316] but for open-ended tasks).
• Learning environment dynamics from interaction traces.

Multi-Agent Ecosystems

The A2A protocol [115] and multi-agent frameworks hint at a future where hundreds of specialized agents collaborate, negotiate, and delegate—forming an “economy of agents” [267]. Open challenges:

• Trust and verification between agents with different principals.
• Emergent cooperation vs. emergent deception in competitive settings [153].
• Market mechanisms for resource allocation (compute, tool access, priority).
• Governance: who is responsible when a chain of 10 agents produces a harmful outcome? [6]

Agent Security and Trust

Autonomous agents inherit every security vulnerability of the LLMs they are built on—plus new attack surfaces created by tool access, multi-agent delegation, and persistent memory (Chapters 19–21). Critical unsolved problems:

• Prompt injection at scale[120]: As agents consume untrusted content (web pages, emails, API responses), indirect prompt injection becomes systemic. No robust defense exists today.
• Confused deputy attacks: An agent with legitimate credentials can be tricked into misusing them on behalf of an attacker embedded in the data stream [10].
• Sandboxing without crippling: Least-privilege execution constrains what agents can do, but overly restrictive sandboxes negate agentic value. Finding the right boundary is an open design problem.
• Audit and attribution: When an agent chain spans multiple organizations (via A2A [115]), tracing who authorized what action remains architecturally unsolved.
• Trust calibration: Agents must learn when not to trust—whether a tool response is authentic, whether another agent’s claim is verified.

Evaluation Beyond Benchmarks

Chapter 14 showed that benchmarks shape the systems we build—yet current evaluation has critical gaps:

• Real-world deployment metrics: Benchmarks like SWE-bench [169] and GAIA [254] measure isolated tasks; production agents face ambiguous goals, shifting requirements, and multi-turn recovery.
• Reward model validity: RLHF assumes reward models capture human preferences, but reward hacking [333] and distributional shift undermine this assumption at scale.
• Cost-quality frontiers: Two agents may achieve the same accuracy, but one costs 10$\times$ more tokens. Evaluation must become cost-aware.
• Safety under distribution shift: An agent safe in testing may behave unsafely on novel inputs. Adversarial evaluation [290] and red-teaming at agentic scale remain immature.

Efficiency and Accessibility

Training a 70B model with RLHF costs $10K--$100K. Running autonomous agents costs $1--$50 per complex task. For agentic AI to achieve broad impact:

• Distillation of agentic capabilities from large to small models [139, 184].
• More efficient RL algorithms (fewer samples, lower variance) [319].
• On-device agents that operate without cloud round-trips.
• Open-weight models that match proprietary quality for agentic tasks [69].

Further Reading

Foundational Papers

• Attention Is All You Need[357] — The transformer architecture.
• RLHF / InstructGPT[280] — The first large-scale RLHF deployment.
• PPO[319] — Proximal Policy Optimization.
• DPO[302] — Direct Preference Optimization.
• GRPO / DeepSeek-R1[323, 69] — Group Relative Policy Optimization and emergent reasoning.
• ReAct[409] — Reasoning + Acting framework for LLM agents.
• Toolformer[315] — Teaching LLMs to use tools.
• RAG[209] — Retrieval-Augmented Generation.

Systems and Scaling

• Megatron-LM[327] — Tensor and pipeline parallelism.
• DeepSpeed ZeRO[304] — Memory-efficient distributed training.
• vLLM[196] — PagedAttention for efficient LLM serving.
• Flash Attention[64] — IO-aware exact attention.

Agentic AI

• Building Effective Agents[9] — Design patterns and principles.
• Voyager[362] — Open-ended agent with skill library in Minecraft.
• SWE-bench[169] — Benchmark for autonomous software engineering.
• OSWorld[397] — Full computer-use benchmarks.
• GAIA[254] — General AI Assistants benchmark for real-world tasks.
• MemGPT[281] — OS-inspired memory management for unbounded context.
• Model Context Protocol[10] — Open standard for tool integration.
• Agent-to-Agent Protocol[115] — Inter-agent communication standard.

Alignment and Safety

• Constitutional AI[16] — Self-supervised alignment.
• Sleeper Agents[153] — Deceptive alignment concerns.
• Reflexion[326] — Learning from verbal self-reflection.
• Indirect Prompt Injection[120] — Security risks for LLM-integrated applications.

Online Resources

• HuggingFace TRL: https://github.com/huggingface/trl — Production RL library.
• LangGraph: https://github.com/langchain-ai/langgraph — Agent workflow graphs.
• OpenAI Agents SDK: https://github.com/openai/openai-agents-python — Official agent framework.
• DeepSpeed-Chat: https://github.com/microsoft/DeepSpeedExamples — End-to-end RLHF pipeline.
• DSPy: https://github.com/stanfordnlp/dspy — Declarative prompt optimization.
• AutoGen: https://github.com/microsoft/autogen — Multi-agent conversation framework.

“The best way to predict the future is to build it.”

— Alan Kay

References

1. [1]A. Abid, A. Abdalla, A. Abid, D. Khan, A. Alfozan, and J. Zou (2019) Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild. External Links: Link
2. [2]S. Abnar and W. Zuidema (2020) Quantifying Attention Flow in Transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link
3. [3]A. Aghajanyan, S. Gupta, and L. Zettlemoyer (2021) Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link
4. [4]M. AI (2025) The Llama 4 Herd: The Beginning of a New Era of Natively Multimodal AI. Meta AI Blog. External Links: Link
5. [5]M. AI (2024) Mistral Large 2. Mistral AI Blog. External Links: Link
6. [6]D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané (2016) Concrete Problems in AI Safety. arXiv Preprint arXiv:1606.06565. External Links: Link
7. [7]Y. An, L. Shen, Y. Xu, and X. Liu (2025) DPPO: Direct Divergence-Based Policy Optimization for Language Model Alignment. arXiv Preprint arXiv:2503.14532. External Links: Link
8. [8]M. Andrychowicz, F. Wolski, A. Ray, et al. (2017) Hindsight Experience Replay. In Advances in Neural Information Processing Systems (NeurIPS),
9. [9]Anthropic (2024) Building Effective Agents. External Links: Link
10. [10]Anthropic (2024) Model Context Protocol. External Links: Link
11. [11]Anthropic (2024) The Claude 3 Model Family: Opus, Sonnet, Haiku. Anthropic Technical Report. External Links: Link
12. [12]Anthropic (2026) Natural Language Autoencoders: Interpreting Neural Networks with Natural Language Descriptions. Anthropic Research Blog. External Links: Link
13. [13]A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi (2023) Self-RAG: Learning to Retrieve, Generate, and Critique Through Self-Reflection. arXiv Preprint arXiv:2310.11511. External Links: Link
14. [14]M. G. Azar, M. Rowland, B. Piot, et al. (2024) A General Theoretical Paradigm to Understand Learning from Human Feedback. arXiv Preprint arXiv:2310.12036. External Links: Link
15. [15]J. L. Ba, J. R. Kiros, and G. E. Hinton (2016) Layer Normalization. arXiv Preprint arXiv:1607.06450. External Links: Link
16. [16]Y. Bai, A. Jones, K. Ndousse, et al. (2022) Constitutional AI: Harmlessness from AI Feedback. arXiv Preprint arXiv:2212.08073. External Links: Link
17. [17]Z. Bai Y. Bao et al. (2025) Kimi K2: Open Agentic Intelligence. arXiv preprint arXiv:2507.20534. External Links: Link
18. [18]P. Bajaj, D. Campos, N. Craswell, et al. (2016) MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. arXiv Preprint arXiv:1611.09268.
19. [19]S. Banerjee and A. Lavie (2005) METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, External Links: Link
20. [20]O. Barkan, E. Hauon, A. Caciularu, I. Dagan, and N. Koenigstein (2021) Grad-SAM: Explaining Transformers via Gradient Self-Attention Maps. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management (CIKM), External Links: Link
21. [21]A. G. Barto and S. Mahadevan (2003) Recent Advances in Hierarchical Reinforcement Learning. Discrete Event Dynamic Systems13, pp. 341–379.
22. [22]F. L. Bellifemine, G. Caire, and D. Greenwood (2007) JADE: Developing Multi-Agent Systems with JADE.
23. [23]I. Beltagy, M. E. Peters, and A. Cohan (2020) Longformer: The Long-Document Transformer. arXiv Preprint arXiv:2004.05150. External Links: Link
24. [24]T. Berners-Lee, J. Hendler, and O. Lassila (2001) The Semantic Web. Scientific American.
25. [25]M. Besta, N. Blach, A. Kubicek, et al. (2024) Graph of Thoughts: Solving Elaborate Problems with Large Language Models. In Proceedings of the AAAI Conference on Artificial Intelligence,
26. [26]L. Beurer-Kellner, M. Fischer, and M. Vechev (2023) Prompting Is Programming: A Query Language for Large Language Models. In Proceedings of PLDI,
27. [27]R. Bonatti, D. Zhao, F. Bonacci, et al. (2024) Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale. arXiv Preprint arXiv:2409.08264. External Links: Link
28. [28]E. d. Bono (1985) Six Thinking Hats.
29. [29]R. A. Bradley and M. E. Terry (1952) Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika. External Links: Link
30. [30]T. Bricken, A. Templeton, J. Batson, et al. (2023) Towards Monosemanticity: Decomposing Language Models with Dictionary Learning. Transformer Circuits Thread. External Links: Link
31. [31]G. Brockman, V. Cheung, L. Pettersson, et al. (2016) OpenAI Gym.
32. [32]T. Brown, B. Mann, N. Ryder, et al. (2020) Language Models Are Few-Shot Learners. NeurIPS.
33. [33]C. Burges, T. Shaked, E. Renshaw, et al. (2005) Learning to Rank Using Gradient Descent. In Proceedings of the 22nd International Conference on Machine Learning (ICML),
34. [34]C. J. C. Burges, R. Ragno, and Q. V. Le (2006) Learning to Rank with Nonsmooth Cost Functions. In Advances in Neural Information Processing Systems (NeurIPS),
35. [35]T. Cai, Y. Li, Z. Geng, et al. (2024) Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. In Proceedings of the 41st International Conference on Machine Learning (ICML), External Links: Link
36. [36]B. Campbell, J. Bradley, N. Sakimura, and T. Lodderstedt (2020) OAuth 2.0 Mutual-TLS Client Authentication and Certificate-Bound Access Tokens. External Links: Link
37. [37]M. Campbell, A. J. H. Jr., and F. Hsu (2002) Deep Blue. Artificial Intelligence.
38. [38]Z. Cao, T. Qin, T. Liu, M. Tsai, and H. Li (2007) Learning to Rank: From Pairwise Approach to Listwise Approach. In Proceedings of the 24th International Conference on Machine Learning (ICML),
39. [39]N. Carlini, F. Tramer, E. Wallace, et al. (2021) Extracting Training Data from Large Language Models. USENIX Security Symposium. External Links: Link
40. [40]Chainlit (2024) Chainlit: Build Production-Ready Conversational AI Applications. External Links: Link
41. [41]H. Chase (2022) LangChain. External Links: Link
42. [42]K. Chawla, J. Ramirez, R. Clever, G. Lucas, J. May, and J. Gratch (2021) CaSiNo: A Corpus of Campsite Negotiation Dialogues for Automatic Negotiation Systems. In Proceedings of NAACL,
43. [43]H. Chen, G. Zheng, Y. Kim, and Y. Chen (2024) Noise Contrastive Alignment of Language Models with Explicit Rewards. arXiv Preprint arXiv:2402.05369. External Links: Link
44. [44]J. Chen, S. Luo, M. Zhang, Z. Liu, Y. Xiao, and D. Han (2022) Hybrid Retrieval for Open-Domain Question Answering. arXiv Preprint arXiv:2210.06029. External Links: Link
45. [45]J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu (2024) BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv Preprint arXiv:2402.03216.
46. [46]L. Chen, M. Zaharia, and J. Zou (2023) FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv Preprint arXiv:2305.05176. External Links: Link
47. [47]M. Chen, J. Tworek, H. Jun, et al. (2021) Evaluating Large Language Models Trained on Code. arXiv Preprint arXiv:2107.03374. External Links: Link
48. [48]S. Chen, S. Wong, L. Chen, and Y. Tian (2023) Extending Context Window of Large Language Models via Positional Interpolation. arXiv Preprint arXiv:2306.15595. External Links: Link
49. [49]T. Chen, B. Xu, C. Zhang, and C. Guestrin (2016) Training Deep Nets with Sublinear Memory Cost. arXiv Preprint arXiv:1604.06174.
50. [50]Y. Chen et al. (2026) MOSAIC: Safe Agentic Training via Plan-Check-Act-Refuse. Note: arXiv preprint
51. [51]Z. Chen, Y. Deng, R. Zhang, H. Sun, and W. Chen (2025) GSPO: Sequence-Level Policy Optimization for Language Model Alignment. arXiv Preprint arXiv:2502.12459. External Links: Link
52. [52]B. Cherny (2026) Loops Are the Next Step from Agents. Note: Talk at Meta @Scale ConferenceHead of Claude Code, Anthropic
53. [53]P. Chhikara, D. Khant, S. Aryan, T. Singh, and D. Yadav (2025) Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. arXiv Preprint arXiv:2504.19413. External Links: Link
54. [54]S. Choi, H. Park, D. Moon, K. Kim, and E. Choi (2025) GOPO: Group Ordinal Policy Optimization for LLM Alignment with Non-Verifiable Rewards. arXiv Preprint arXiv:2505.12948. External Links: Link
55. [55]A. Chowdhery, S. Narang, J. Devlin, et al. (2022) PaLM: Scaling Language Modeling with Pathways. arXiv Preprint arXiv:2204.02311.
56. [56]S. R. Chowdhury, A. Chakraborty, S. Natarajan, A. Agarwal, and D. Sontag (2024) Provably Robust DPO: Aligning Language Models with Noisy Feedback. arXiv Preprint arXiv:2403.00409. External Links: Link
57. [57]P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei (2017) Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
58. [58]Y. Chuang, Y. Xie, H. Luo, Y. Kim, J. Glass, and P. He (2024) DoLA: Decoding by Contrasting Layers Improves Factuality in Large Language Models. In Proceedings of the 12th International Conference on Learning Representations (ICLR), External Links: Link
59. [59]H. W. Chung, L. Hou, S. Longpre, et al. (2024) Scaling Instruction-Finetuned Language Models. Journal of Machine Learning Research. External Links: Link
60. [60]J. Cohen (1960) A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement.
61. [61]W. W. W. Consortium (2022) Decentralized Identifiers (DIDs) v1.0. External Links: Link
62. [62]G. V. Cormack, C. L. A. Clarke, and S. Buettcher (2009) Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. In Proceedings of the 32nd International ACM SIGIR Conference,
63. [63]H. Cunningham, A. Ewart, L. Riggs, R. Huben, and L. Sharkey (2024) Sparse Autoencoders Find Highly Interpretable Features in Language Models. In Proceedings of the 12th International Conference on Learning Representations (ICLR), External Links: Link
64. [64]T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré (2022) FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
65. [65]T. Dao (2024) FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. In International Conference on Learning Representations (ICLR), External Links: Link
66. [66]A. Das, T. Gerber, G. Gkioxari, S. Lee, D. Parikh, and D. Batra (2019) TarMAC: Targeted Multi-Agent Communication. In Proceedings of the 36th International Conference on Machine Learning (ICML), External Links: Link
67. [67]Databricks (2024) DBRX: A New State-of-the-Art Open LLM. Databricks Blog. External Links: Link
68. [68]G. DeepMind (2024) AlphaProof and AlphaGeometry 2: Solving Olympiad Geometry Without Human Demonstrations. External Links: Link
69. [69]D. G. DeepSeek-AI D. Yang et al. (2025) DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv Preprint arXiv:2501.12948. External Links: Link
70. [70]DeepSeek-AI (2024) DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv Preprint arXiv:2405.04434. External Links: Link
71. [71]DeepSeek-AI (2024) DeepSeek-V3 Technical Report. arXiv Preprint arXiv:2412.19437. External Links: Link
72. [72]DeepSeek-AI (2026) DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv preprint arXiv:2606.19348. External Links: Link
73. [73]X. Deng, Y. Gu, B. Zheng, et al. (2023) Mind2Web: Towards a Generalist Agent for the Web. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
74. [74]M. Dennis, N. Jaques, E. Vinitsky, et al. (2020) Emergent Complexity and Zero-Shot Transfer via Unsupervised Environment Design. In Advances in Neural Information Processing Systems (NeurIPS),
75. [75]T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer (2023) QLoRA: Efficient Finetuning of Quantized Language Models. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
76. [76]J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT, External Links: Link
77. [77]Y. Dong, C. Moon, Y. Wang, et al. (2024) XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models. arXiv Preprint arXiv:2411.15100.
78. [78]Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch (2023) Improving Factuality and Reasoning in Language Models Through Multiagent Debate. In Proceedings of the 41st International Conference on Machine Learning (ICML), External Links: Link
79. [79]Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel (2016) RL2: Fast Reinforcement Learning via Slow Reinforcement Learning. arXiv Preprint arXiv:1611.02779.
80. [80]E. H. Durfee, V. R. Lesser, and D. D. Corkill (1989) Trends in Cooperative Distributed Problem Solving. IEEE Transactions on Knowledge and Data Engineering. External Links: Link
81. [81]H. Ebbinghaus (1885) Über Das Gedächtnis: Untersuchungen Zur Experimentellen Psychologie.
82. [82]D. Edge, H. Trinh, N. Cheng, et al. (2024) From Local to Global: A Graph RAG Approach to Query-Focused Summarization. arXiv Preprint arXiv:2404.16130. External Links: Link
83. [83]V. Egiazarian, A. Panferov, D. Kuznedelev, E. Frantar, A. Babber, and D. Alistarh (2024) Extreme Compression of Large Language Models via Additive Quantization. arXiv Preprint arXiv:2401.06118. External Links: Link
84. [84]A. E. Elo (1978) The Rating of Chess Players, Past and Present.
85. [85]S. Es, J. James, L. Espinosa-Anke, and S. Schockaert (2023) RAGAs: Automated Evaluation of Retrieval Augmented Generation. arXiv Preprint arXiv:2309.15217. External Links: Link
86. [86]K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela (2024) KTO: Model Alignment as Prospect Theoretic Optimization. arXiv Preprint arXiv:2402.01306. External Links: Link
87. [87]K. Ethayarajh (2019) How Contextual Are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), External Links: Link
88. [88]H. Face (2025) OpenEnv: An Interface Library for RL Post-Training with Environments. External Links: Link
89. [89]W. Fedus, B. Zoph, and N. Shazeer (2022) Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Journal of Machine Learning Research.
90. [90]D. Ferrucci, E. Brown, J. Chu-Carroll, et al. (2010) Building Watson: An Overview of the DeepQA Project. AI Magazine.
91. [91]J. R. Firth (1957) A Synopsis of Linguistic Theory, 1930–1955. Studies in Linguistic Analysis.
92. [92]J. L. Fleiss (1971) Measuring Nominal Scale Agreement Among Many Raters. Psychological Bulletin.
93. [93]F. for Intelligent Physical Agents (2002) FIPA ACL Message Structure Specification. External Links: Link
94. [94]T. Formal, C. Lassance, B. Piwowarski, and S. Clinchant (2021) SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval. arXiv Preprint arXiv:2109.10086.
95. [95]T. Formal, B. Piwowarski, and S. Clinchant (2021) SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval,
96. [96]T. Forte (2022) Building a Second Brain: A Proven Method to Organize Your Digital Life and Unlock Your Creative Potential.
97. [97]N. Francis, A. Green, P. Guagliardo, et al. (2018) Cypher: An Evolving Query Language for Property Graphs. In Proceedings of the 2018 International Conference on Management of Data (SIGMOD),
98. [98]E. Frantar and D. Alistarh (2023) SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. In Proceedings of the 40th International Conference on Machine Learning (ICML), External Links: Link
99. [99]E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh (2023) GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformers. arXiv Preprint arXiv:2210.17323. External Links: Link
100. [100]Y. Fu, R. Panda, X. Niu, et al. (2024) Data Engineering for Scaling Language Models to 128K Context. arXiv Preprint arXiv:2402.10171. External Links: Link
101. [101]Y. Fu, P. Bailis, I. Stoica, and H. Zhang (2024) Break the Sequential Dependency of LLM Inference Using Lookahead Decoding. arXiv Preprint arXiv:2402.02057. External Links: Link
102. [102]P. Furgale, S. Klingler, J. Nolan, M. Staats, G. Di Lorenzo, E. Martinez Abad, C. Schueler, R. Dinu, A. Devoto, P. Berard, G. Kaplun, E. Sarafian, R. Roveri, L. Derczynski, and R. Silveira Cabral (2026) NVIDIA-labs OO Agents: Native Python Object-Oriented Agents. Technical report Note: arXiv preprint arXiv:2607.20709 External Links: Link
103. [103]A. Gal, A. Anaby-Tavor, A. Trombetta, and D. Montesi (2005) A Framework for Modeling and Evaluating Automatic Semantic Reconciliation. In Proceedings of the 31st International Conference on Very Large Data Bases (VLDB), External Links: Link
104. [104]I. O. Gallegos, R. A. Rossi, J. Barber, S. Tong, et al. (2024) Bias and Fairness in Large Language Models: A Survey. Computational Linguistics. External Links: Link
105. [105]L. Gao, J. Schulman, and J. Hilton (2023) Scaling Laws for Reward Model Overoptimization. In Proceedings of the 40th International Conference on Machine Learning (ICML),
106. [106]L. Gao, X. Ma, J. Lin, and J. Callan (2023) Precise Zero-Shot Dense Retrieval Without Relevance Labels. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link
107. [107]J. García and F. Fernández (2015) A Comprehensive Survey on Safe Reinforcement Learning. Journal of Machine Learning Research.
108. [108]G. Gemini Team (2024) Gemini 1.5: Unlocking Multimodal Understanding Across Millions of Tokens of Context. arXiv Preprint arXiv:2403.05530. External Links: Link
109. [109]G. Gerganov (2023) GGUF: GPT-Generated Unified Format (llama.cpp). External Links: Link
110. [110]M. Geva, R. Schuster, J. Berant, and O. Levy (2021) Transformer Feed-Forward Layers Are Key-Value Memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP),
111. [111]F. Gloeckle, B. Y. Idrissi, B. Rozière, D. Lopez-Paz, and G. Synnaeve (2024) Better & Faster Large Language Models via Multi-Token Prediction. In Proceedings of the 41st International Conference on Machine Learning (ICML), External Links: Link
112. [112]C. A. E. Goodhart (1984) Problems of Monetary Management: The U.K. Experience. Monetary Theory and Practice.
113. [113]Google DeepMind (2026) Decoupled DiLoCo: Distributed Training Across Geographic Regions. Note: arXiv preprint
114. [114]Google Research (2026) Thinking to Recall: How Reasoning Unlocks Parametric Knowledge in LLMs. Note: COLM 2026
115. [115]Google (2025) Agent2Agent (A2A) Protocol. External Links: Link
116. [116]A. Gorbatenko (2024) Online DPO with Synchronised Reference Model Updates. TRL Documentation.
117. [117]P. Grassé (1959) La Reconstruction Du Nid Et Les Coordinations Interindividuelles Chez Bellicositermes Natalensis Et Cubitermes Sp. La Théorie de La Stigmergie: Essai d’interprétation Du Comportement Des Termites Constructeurs. Insectes Sociaux.
118. [118]A. Grattafiori, A. Dubey, A. Jauhri, et al. (2024) The Llama 3 Herd of Models. arXiv Preprint arXiv:2407.21783. External Links: Link
119. [119]A. Graves, G. Wayne, M. Reynolds, et al. (2016) Hybrid Computing Using a Neural Network with Dynamic External Memory. Nature.
120. [120]K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz (2023) Not What You’ve Signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security (AISec),
121. [121]J. W. Group (2010) JSON-RPC 2.0 Specification. External Links: Link
122. [122]A. Gu and T. Dao (2023) Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv Preprint arXiv:2312.00752. External Links: Link
123. [123]A. Gu and T. Dao (2024) Mamba: Linear-Time Sequence Modeling with Selective State Spaces. In Proceedings of the 41st International Conference on Machine Learning (ICML), External Links: Link
124. [124]C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger (2017) On Calibration of Modern Neural Networks. In International Conference on Machine Learning (ICML),
125. [125]M. Guo, J. Ainslie, D. Uthus, et al. (2022) LongT5: Efficient Text-to-Text Transformer for Long Sequences. Findings of the Association for Computational Linguistics: NAACL 2022. External Links: Link
126. [126]R. Guo, P. Sun, E. Lindgren, et al. (2020) Accelerating Large-Scale Inference with Anisotropic Vector Quantization. In International Conference on Machine Learning (ICML),
127. [127]S. Guo, B. Zhang, T. Liu, et al. (2024) Direct Language Model Alignment from Online AI Feedback. arXiv Preprint arXiv:2402.04792. External Links: Link
128. [128]K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang (2020) REALM: Retrieval-Augmented Language Model Pre-Training. In Proceedings of the 37th International Conference on Machine Learning (ICML), External Links: Link
129. [129]T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine (2018) Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In Proceedings of the 35th International Conference on Machine Learning (ICML), External Links: Link
130. [130]D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi (2020) Dream to Control: Learning Behaviors by Latent Imagination. In Proceedings of the 8th International Conference on Learning Representations (ICLR), External Links: Link
131. [131]C. Han, M. Li, and W. Chen (2025) SAPO: Soft Adaptive Policy Optimization for Efficient LLM Alignment. arXiv Preprint arXiv:2503.01739. External Links: Link
132. [132]D. Han and M. Han (2024) Unsloth: Efficient LLM Fine-Tuning. External Links: Link
133. [133]D. Hardt (2012) The OAuth 2.0 Authorization Framework. External Links: Link
134. [134]S. Harris and A. Seaborne (2013) SPARQL 1.1 Query Language. External Links: Link
135. [135]B. Hayes-Roth (1985) A Blackboard Architecture for Control. Artificial Intelligence. External Links: Link
136. [136]S. Hayou, N. Ghosh, and B. Yu (2024) LoRA+: Efficient Low Rank Adaptation of Large Models. arXiv Preprint arXiv:2402.12354. External Links: Link
137. [137]D. Hendrycks, C. Burns, S. Basart, et al. (2021) Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations (ICLR),
138. [138]R. Herbrich, T. Minka, and T. Graepel (2006) TrueSkill™: A Bayesian Skill Rating System. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
139. [139]G. Hinton, O. Vinyals, and J. Dean (2015) Distilling the Knowledge in a Neural Network. arXiv Preprint arXiv:1503.02531.
140. [140]J. Hoffmann, S. Borgeaud, A. Mensch, et al. (2022) Training Compute-Optimal Large Language Models. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
141. [141]J. Hong, N. Lee, and J. Thorne (2024) ORPO: Monolithic Preference Optimization Without Reference Model. arXiv Preprint arXiv:2403.07691. External Links: Link
142. [142]S. Hong, M. Zhuge, J. Chen, et al. (2024) MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework.
143. [143]A. Hosseini, X. Yuan, P. Poupart, A. Trischler, and Y. Bengio (2024) V-STaR: Training Verifiers for Self-Taught Reasoners. arXiv Preprint arXiv:2402.06457.
144. [144]N. Houlsby, A. Giber, S. Jastrzebski, et al. (2019) Parameter-Efficient Transfer Learning for NLP. In International Conference on Machine Learning (ICML), External Links: Link
145. [145]P. Hsu, Y. Dai, V. Kothapalli, et al. (2024) Liger-Kernel: Efficient Triton Kernels for LLM Training. arXiv Preprint arXiv:2410.10989. External Links: Link
146. [146]E. J. Hu, Y. Shen, P. Wallis, et al. (2022) LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations (ICLR), External Links: Link
147. [147]E. J. Hu, Y. Shen, P. Wallis, et al. (2022) LoRA: Low-Rank Adaptation of Large Language Models. arXiv Preprint arXiv:2106.09685.
148. [148]J. Hu, X. Tao, W. Zhu, S. Yang, J. Liu, and Z. Li (2024) OpenRLHF: An Easy-to-Use, Scalable and High-Performance RLHF Framework. arXiv Preprint arXiv:2405.11143.
149. [149]Q. Hu S. Yang et al. (2025) Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter. arXiv Preprint arXiv:2511.16665. External Links: Link
150. [150]S. Hu, Y. Tu, X. Han, et al. (2024) MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies. arXiv Preprint arXiv:2404.06395. External Links: Link
151. [151]Q. Huang, J. Vora, P. Liang, and J. Leskovec (2024) MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation. In Proceedings of the 41st International Conference on Machine Learning (ICML), External Links: Link
152. [152]Y. Huang, Y. Cheng, A. Bapna, et al. (2019) GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism. In Advances in Neural Information Processing Systems (NeurIPS),
153. [153]E. Hubinger, C. Denison, J. Mu, et al. (2024) Sleeper Agents: Training Deceptive LLMs That Persist Through Safety Training. arXiv Preprint arXiv:2401.05566.
154. [154]L. Inc (2024) LangGraph Studio: The First Agent IDE. External Links: Link
155. [155]L. Inc (2024) LangGraph: Build Stateful Multi-Actor Applications with LLMs. External Links: Link
156. [156]S. Inc (2024) Streamlit: The Fastest Way to Build and Share Data Apps. External Links: Link
157. [157]G. Irving, P. Christiano, and D. Amodei (2018) AI Safety via Debate. arXiv Preprint arXiv:1805.00899. External Links: Link
158. [158]M. Jaderberg, W. M. Czarnecki, I. Dunning, et al. (2019) Human-Level Performance in 3D Multiplayer Games with Population-Based Reinforcement Learning. Science. External Links: Link
159. [159]N. Jain, P. Chiang, Y. Wen, et al. (2024) NEFTune: Noisy Embeddings Improve Instruction Finetuning. In International Conference on Learning Representations (ICLR), External Links: Link
160. [160]S. Jain and B. C. Wallace (2019) Attention Is Not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), External Links: Link
161. [161]E. Jang, S. Gu, and B. Poole (2017) Categorical Reparameterization with Gumbel-Softmax. In International Conference on Learning Representations (ICLR),
162. [162]S. Jeong, J. Baek, S. Cho, S. J. Hwang, and J. C. Park (2024) Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models Through Question Complexity. arXiv Preprint arXiv:2403.14403. External Links: Link
163. [163]J. Ji, A. Fisch, J. Weston, and S. Sukhbaatar (2024) Towards Exact Optimization of Language Model Alignment. arXiv Preprint arXiv:2402.05369. External Links: Link
164. [164]Z. Ji, N. Lee, R. Frieske, et al. (2023) Survey of Hallucination in Natural Language Generation. ACM Computing Surveys. External Links: Link
165. [165]A. Q. Jiang, A. Sablayrolles, A. Mensch, et al. (2023) Mistral 7B. arXiv Preprint arXiv:2310.06825. External Links: Link
166. [166]A. Q. Jiang, A. Sablayrolles, A. Roux, et al. (2024) Mixtral of Experts. arXiv Preprint arXiv:2401.04088. External Links: Link
167. [167]M. Jiang, E. Grefenstette, and T. Rocktäschel (2021) Prioritized Level Replay. In Proceedings of the 38th International Conference on Machine Learning (ICML), External Links: Link
168. [168]Y. Jiang et al. (2025) Reasoning Beyond Chain-of-Thought: A Latent Computational View. Note: arXiv preprint arXiv:2601.08058 External Links: Link
169. [169]C. E. Jimenez, J. Yang, A. Wettig, et al. (2024) SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. In International Conference on Learning Representations (ICLR), External Links: Link
170. [170]B. Jin H. Zeng et al. (2025) Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. arXiv Preprint arXiv:2503.09516.
171. [171]J. Johnson, M. Douze, and H. Jégou (2021) Billion-Scale Similarity Search with GPUs. IEEE Transactions on Big Data.
172. [172]M. Jones, J. Bradley, and N. Sakimura (2015) JSON Web Token (JWT). External Links: Link
173. [173]M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer (2017) TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings of ACL,
174. [174]J. Jumper, R. Evans, A. Pritzel, et al. (2021) Highly Accurate Protein Structure Prediction with AlphaFold. Nature.
175. [175]S. Kadavath, T. Conerly, A. Askell, et al. (2022) Language Models (Mostly) Know What They Know. arXiv Preprint arXiv:2207.05221. External Links: Link
176. [176]D. Kalajdzievski (2023) A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA. arXiv Preprint arXiv:2312.03732. External Links: Link
177. [177]J. Kaplan, S. McCandlish, T. Henighan, et al. (2020) Scaling Laws for Neural Language Models. arXiv Preprint arXiv:2001.08361. External Links: Link
178. [178]A. Karpathy (2025) Context Engineering for AI Agents. Note: Social media postJune 2025
179. [179]A. Karpathy (2026) AutoResearch: Automated ML Experimentation via Agent Loops. Note: Open-source project External Links: Link
180. [180]V. Karpukhin, B. Oğuz, S. Min, et al. (2020) Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), External Links: Link
181. [181]J. Ke, Z. Wen, W. Li, C. He, and L. Zhang (2026) Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning. arXiv preprint arXiv:2605.13255. External Links: Link
182. [182]O. Khattab, A. Singhvi, P. Maheshwari, et al. (2024) DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. In Proceedings of the 12th International Conference on Learning Representations (ICLR), External Links: Link
183. [183]O. Khattab and M. Zaharia (2020) ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. In Proceedings of the 43rd International ACM SIGIR Conference, External Links: Link
184. [184]S. Kim S. J. Jang et al. (2023) Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. Findings of the ACL. External Links: Link
185. [185]D. P. Kingma and J. Ba (2015) Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR), External Links: Link
186. [186]J. Kirkpatrick, R. Pascanu, N. Rabinowitz, et al. (2017) Overcoming Catastrophic Forgetting in Neural Networks. In Proceedings of the National Academy of Sciences,
187. [187]L. Kocsis and C. Szepesvári (2006) Bandit Based Monte-Carlo Planning. In European Conference on Machine Learning (ECML),
188. [188]J. Y. Koh, R. Lo, L. Jang, et al. (2024) VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link
189. [189]T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa (2022) Large Language Models Are Zero-Shot Reasoners. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
190. [190]D. J. Kopiczko, T. Blankevoort, and M. Nagel (2024) VeRA: Vector-Based Random Matrix Adaptation. arXiv Preprint arXiv:2310.11454. External Links: Link
191. [191]V. A. Korthikanti, J. Casper, S. Lym, et al. (2023) Reducing Activation Recomputation in Large Transformer Models. In Proceedings of Machine Learning and Systems (MLSys),
192. [192]A. Krizhevsky, I. Sutskever, and G. E. Hinton (2012) ImageNet Classification with Deep Convolutional Neural Networks. NeurIPS.
193. [193]L. Kuhn, Y. Gal, and S. Farquhar (2023) Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation. In International Conference on Learning Representations (ICLR), External Links: Link
194. [194]H. Küttler, N. Nardelli, A. H. Miller, et al. (2020) The NetHack Learning Environment. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
195. [195]T. Kwiatkowski, J. Palomaki, O. Redfield, et al. (2019) Natural Questions: A Benchmark for Question Answering Research. Transactions of the Association for Computational Linguistics.
196. [196]W. Kwon, Z. Li, S. Zhuang, et al. (2023) Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles (SOSP), External Links: Link
197. [197]T. Lacroix, G. Obozinski, and N. Usunier (2020) Tensor Decompositions for Temporal Knowledge Base Completion. In International Conference on Learning Representations (ICLR), External Links: Link
198. [198]J. Lála, O. O’Donoghue, A. Shtedritski, S. Cox, S. G. Rodriques, and A. D. White (2023) PaperQA: Retrieval-Augmented Generative Agent for Scientific Research. arXiv Preprint arXiv:2312.07559. External Links: Link
199. [199]N. Lambert, J. Morrison, V. Pyatkin, et al. (2024) Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv Preprint arXiv:2411.15124. External Links: Link
200. [200]A. Lazaridou and M. Baroni (2020) Emergent Multi-Agent Communication in the Deep Learning Era. arXiv Preprint arXiv:2006.02419. External Links: Link
201. [201]H. Le, Y. Wang, A. D. Gotmare, S. Savarese, and S. Hoi (2023) CodeRL: Mastering Code Generation Through Pretrained Models and Deep Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS),
202. [202]T. Le Sellier De Chezelles, M. Gasse, A. Drouin, M. Caccia, et al. (2024) The BrowserGym Ecosystem for Web Agent Research. arXiv Preprint arXiv:2412.05467. External Links: Link
203. [203]C. Lee, R. Roy, M. Xu, et al. (2024) NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models. arXiv Preprint arXiv:2405.17428.
204. [204]K. Lee, D. Ippolito, A. Nystrom, et al. (2022) Deduplicating Training Data Makes Language Models Better. In Proceedings of the 60th Annual Meeting of the ACL, External Links: Link
205. [205]B. Lester, R. Al-Rfou, and N. Constant (2021) The Power of Scale for Parameter-Efficient Prompt Tuning. In Proceedings of the 2021 Conference on EMNLP, External Links: Link
206. [206]Y. Leviathan, M. Kalman, and Y. Matias (2023) Fast Inference from Transformers via Speculative Decoding. In Proceedings of the 40th International Conference on Machine Learning (ICML), External Links: Link
207. [207]M. Lewis, Y. Liu, N. Goyal, et al. (2020) BART: Denoising Sequence-to-Sequence Pre-Training for Natural Language Generation, Translation, and Comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link
208. [208]M. Lewis, D. Yarats, Y. N. Dauphin, D. Parikh, and D. Batra (2017) Deal or No Deal? End-to-End Learning for Negotiation Dialogues. In Proceedings of EMNLP,
209. [209]P. Lewis, E. Perez, A. Piktus, et al. (2020) Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
210. [210]S. Li, Y. Zhao, R. Varma, et al. (2020) PyTorch Distributed: Experiences on Accelerating Data Parallel Training. In Proceedings of the VLDB Endowment,
211. [211]X. Li, A. Holtzman, D. Fried, et al. (2023) Contrastive Decoding: Open-Ended Text Generation as Optimization. ACL.
212. [212]X. L. Li and P. Liang (2021) Prefix-Tuning: Optimizing Continuous Prompts for Generation. In Proceedings of the 59th Annual Meeting of the ACL, External Links: Link
213. [213]Y. Li, F. Wei, C. Zhang, and H. Zhang (2024) EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. In Proceedings of the 41st International Conference on Machine Learning (ICML), External Links: Link
214. [214]Z. Li, X. Zhang, Y. Zhang, D. Long, P. Xie, and M. Zhang (2023) Towards General Text Embeddings with Multi-Stage Contrastive Learning. arXiv Preprint arXiv:2308.03281.
215. [215]Z. Li et al. (2026) UniClawBench: A Bilingual Benchmark for Interactive Agent Evaluation. Note: arXiv preprint
216. [216]H. Lightman, V. Kosaraju, Y. Burda, et al. (2023) Let’s Verify Step by Step. arXiv Preprint arXiv:2305.20050.
217. [217]C. Lin (2004) ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, External Links: Link
218. [218]J. Lin, J. Tang, H. Tang, et al. (2024) AWQ: Activation-Aware Weight Quantization for LLM Compression and Acceleration. In Proceedings of Machine Learning and Systems (MLSys), External Links: Link
219. [219]K. Lin, C. Snell, Y. Wang, et al. (2025) Sleep-Time Compute: Beyond Inference Scaling at Test-Time. arXiv Preprint arXiv:2504.13171. External Links: Link
220. [220]L. Lin (1992) Self-Improving Reactive Agents Based on Reinforcement Learning, Planning and Teaching. Machine Learning.
221. [221]X. Lin, A. Ghosh, B. K. H. Low, A. Shrivastava, and V. Mohan (2025) REFRAG: Rethinking RAG Based Decoding. arXiv Preprint arXiv:2509.01092.
222. [222]H. Liu, M. Zaharia, and P. Abbeel (2023) Ring Attention with Blockwise Transformers for Near-Infinite Context. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
223. [223]H. Liu, D. Tam, M. Muqeeth, et al. (2022) Few-Shot Parameter-Efficient Fine-Tuning Is Better and Cheaper Than in-Context Learning. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
224. [224]J. Liu, D. Shen, Y. Zhang, B. Dolan, L. Carin, and W. Chen (2022) What Makes Good in-Context Examples for GPT-3?. In Proceedings of Deep Learning Inside Out (DeeLIO), ACL Workshop, External Links: Link
225. [225]K. J. Liu, Y. Jin, Z. Zhou, et al. (2025) Muon: An Optimizer for Hidden Layers in Neural Networks. Note: arXiv preprint External Links: Link
226. [226]N. F. Liu, K. Lin, J. Hewitt, et al. (2024) Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics. External Links: Link
227. [227]N. F. Liu, K. Lin, J. Hewitt, et al. (2024) Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics. External Links: Link
228. [228]Q. e. a. Liu (2025) Answer Convergence as a Signal for Early Stopping in Reasoning. In Proceedings of EMNLP, External Links: Link
229. [229]S. Liu, C. Wang, H. Yin, et al. (2024) DoRA: Weight-Decomposed Low-Rank Adaptation. arXiv Preprint arXiv:2402.09353. External Links: Link
230. [230]X. Liu, H. Yu, H. Zhang, et al. (2023) AgentBench: Evaluating LLMs as Agents. arXiv Preprint arXiv:2308.03688.
231. [231]X. e. a. Liu (2025) Learning Like Humans: Advancing LLM Reasoning Capabilities via Adaptive Difficulty Curriculum Learning. arXiv Preprint arXiv:2505.08364. External Links: Link
232. [232]Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu (2023) G-Eval: NLG Evaluation Using GPT-4 with Better Human Alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), External Links: Link
233. [233]Y. Liu, L. Han, Y. Tan, et al. (2025) Understanding and Mitigating the Pretraining Distribution Bias in GRPO. arXiv Preprint arXiv:2505.07888. External Links: Link
234. [234]Y. Liu, M. Ott, N. Goyal, et al. (2019) RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv Preprint arXiv:1907.11692. External Links: Link
235. [235]Z. Liu, B. Oguz, C. Zhao, et al. (2023) LLM-QAT: Data-Free Quantization Aware Training for Large Language Models. arXiv Preprint arXiv:2305.17888. External Links: Link
236. [236]Z. Liu H. Dong et al. (2025) DLER: Doing Length Penalty Right—Incentivizing More Intelligence Per Token. arXiv Preprint arXiv:2510.15110. External Links: Link
237. [237]Z. Liu et al. (2025) OctoThinker: Mid-Training Determines RL-Readiness of Base Models. Note: arXiv preprint
238. [238]Z. Liu, J. Yuan, H. Jin, et al. (2024) KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. In International Conference on Machine Learning (ICML), External Links: Link
239. [239]I. Loshchilov and F. Hutter (2019) Decoupled Weight Decay Regularization. arXiv Preprint arXiv:1711.05101. External Links: Link
240. [240]R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch (2017) Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
241. [241]C. Lu, C. Lu, R. T. Lange, J. Foerster, J. Clune, and D. Ha (2024) The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. arXiv Preprint arXiv:2408.06292. External Links: Link
242. [242]Y. Lu, M. Bartolo, A. Moore, S. Riedel, and P. Stenetorp (2022) Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link
243. [243]Z. Luo, Z. Chen, Y. Jiang, et al. (2025) ScaleRL: Scaling Reinforcement Learning for LLM Reasoning. arXiv Preprint arXiv:2505.16356. External Links: Link
244. [244]Z. Luo, J. Shi, T. Yu, and M. Chen (2025) VESPO: Variational Sequence-Level Soft Policy Optimization for LLM Alignment. arXiv Preprint arXiv:2505.07508. External Links: Link
245. [245]T. Lütke (2025) Context Engineering. Note: Social media postCEO of Shopify, June 2025
246. [246]A. Madaan, N. Tandon, P. Gupta, et al. (2023) Self-Refine: Iterative Refinement with Self-Feedback. In Advances in Neural Information Processing Systems (NeurIPS),
247. [247]A. Maharana, D. Lee, S. Tuber, M. Ruber, F. Barbieri, and M. Bansal (2024) LoCoMo: Long-Context Conversation with Memory Operations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP),
248. [248]Yu. A. Malkov and D. A. Yashunin (2020) Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence.
249. [249]P. Manakul, A. Liusie, and M. J. F. Gales (2023) SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models. In Proceedings of EMNLP, External Links: Link
250. [250]S. McCandlish, J. Kaplan, D. Amodei, and O. D. Team (2018) An Empirical Model of Large-Batch Training. arXiv Preprint arXiv:1812.06162.
251. [251]J. e. a. Mei (2025) APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-Tail Generation. arXiv Preprint arXiv:2509.18521. External Links: Link
252. [252]Y. Meng, M. Xia, and D. Chen (2024) SimPO: Simple Preference Optimization with a Reference-Free Reward. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
253. [253]Meta AI (2026) Remember When It Matters: Proactive Memory for Long-Horizon Agents. Note: arXiv preprint
254. [254]G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y. LeCun, and T. Scialom (2024) GAIA: A Benchmark for General AI Assistants. In International Conference on Learning Representations (ICLR), External Links: Link
255. [255]P. Micikevicius, S. Narang, J. Alben, et al. (2018) Mixed Precision Training. In International Conference on Learning Representations (ICLR),
256. [256]Microsoft (2023) Semantic Kernel: SDK for Integrating AI Models into Applications. External Links: Link
257. [257]S. Min, X. Lyu, A. Holtzman, et al. (2022) Rethinking the Role of Demonstrations: What Makes in-Context Learning Work?. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), External Links: Link
258. [258]V. Mnih, A. P. Badia, M. Mirza, et al. (2016) Asynchronous Methods for Deep Reinforcement Learning. In Proceedings of the 33rd International Conference on Machine Learning (ICML), External Links: Link
259. [259]V. Mnih, K. Kavukcuoglu, D. Silver, et al. (2015) Human-Level Control Through Deep Reinforcement Learning. Nature. External Links: Link
260. [260]MosaicML (2023) MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs. MosaicML Blog. External Links: Link
261. [261]J. Moura (2023) CrewAI: Framework for Orchestrating Role-Playing Autonomous AI Agents. External Links: Link
262. [262]R. Nakano, J. Hilton, S. Balaji, et al. (2021) WebGPT: Browser-Assisted Question-Answering with Human Feedback. arXiv Preprint arXiv:2112.09332. External Links: Link
263. [263]D. Narayanan, A. Harlap, A. Phanishayee, et al. (2019) PipeDream: Generalized Pipeline Parallelism for DNN Training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP),
264. [264]D. Narayanan, M. Shoeybi, J. Casper, et al. (2021) Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM. arXiv Preprint arXiv:2104.04473.
265. [265]A. Y. Ng, D. Harada, and S. J. Russell (1999) Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping. In Proceedings of the 16th International Conference on Machine Learning (ICML),
266. [266]M. Nguyen (2024) Min-p Sampling: A Simple Baseline for Better LLM Decoding. arXiv Preprint arXiv:2310.06022.
267. [267]N. Nisan, T. Roughgarden, É. Tardos, and V. V. Vazirani (2007) Algorithmic Game Theory. External Links: Link
268. [268]R. Nogueira and K. Cho (2019) Passage Re-Ranking with BERT. arXiv Preprint arXiv:1901.04085.
269. [269]R. Nogueira, Z. Jiang, R. Pradeep, and J. Lin (2020) Document Ranking with a Pretrained Sequence-to-Sequence Model. Findings of EMNLP.
270. [270]NVIDIA (2025) Dynamo: A Datacenter-Scale Distributed Inference Serving Framework. Note: Open-source project External Links: Link
271. [271]NVIDIA (2025) NeMo Gym: Interactive RL Environments for LLM Agent Training. Note: Technical report External Links: Link
272. [272]M. Nye, A. J. Andreassen, G. Gur-Ari, et al. (2021) Show Your Work: Scratchpads for Intermediate Computation with Language Models. arXiv Preprint arXiv:2112.00114. External Links: Link
273. [273]C. Olsson, N. Elhage, N. Nanda, et al. (2022) In-Context Learning and Induction Heads. Transformer Circuits Thread. External Links: Link
274. [274]OpenAI (2023) GPT-4 Technical Report. arXiv Preprint arXiv:2303.08774.
275. [275]OpenAI (2024) Learning to Reason with LLMs. External Links: Link
276. [276]OpenAI (2024) Swarm: An Educational Framework for Lightweight Multi-Agent Orchestration. External Links: Link
277. [277]OpenAI (2025) OpenAI Agents SDK. External Links: Link
278. [278]OpenAI (2025) OpenAI o3 and o4-mini System Card. External Links: Link
279. [279]A. Osmani (2026) Loop Engineering. Note: Blog post External Links: Link
280. [280]L. Ouyang, J. Wu, X. Jiang, et al. (2022) Training Language Models to Follow Instructions with Human Feedback. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
281. [281]C. Packer, V. Fang, S. G. Patil, K. Lin, S. Wooders, and J. E. Gonzalez (2023) MemGPT: Towards LLMs as Operating Systems. arXiv Preprint arXiv:2310.08560. External Links: Link
282. [282]L. Pan, S. Tao, Y. Zhai, L. Zhang, Z. Liu, B. Ding, A. Liu, and L. Wen (2026) RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation. arXiv preprint arXiv:2606.11709. External Links: Link
283. [283]K. Papineni, S. Roukos, T. Ward, and W. Zhu (2002) BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link
284. [284]R. Parasuraman and V. Riley (1997) Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors. External Links: Link
285. [285]J. S. Park, J. C. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein (2023) Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST), External Links: Link
286. [286]D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell (2017) Curiosity-Driven Exploration by Self-Supervised Prediction. In Proceedings of the 34th International Conference on Machine Learning (ICML),
287. [287]S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez (2024) Gorilla: Large Language Model Connected with Massive APIs. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
288. [288]B. Peng, E. Alcaide, Q. Anthony, et al. (2023) RWKV: Reinventing RNNs for the Transformer Era. Findings of the Association for Computational Linguistics: EMNLP 2023. External Links: Link
289. [289]B. Peng, J. Quesnelle, H. Fan, and E. Shao (2023) YaRN: Efficient Context Window Extension of Large Language Models. arXiv Preprint arXiv:2309.00071.
290. [290]E. Perez, S. Huang, F. Song, et al. (2022) Red Teaming Language Models with Language Models. In Proceedings of EMNLP, External Links: Link
291. [291]R. L. Plackett (1975) The Analysis of Permutations. Journal of the Royal Statistical Society: Series C (Applied Statistics).
292. [292]O. Press, N. A. Smith, and M. Lewis (2022) Train Short, Test Long: Attention with Linear Biases Enables Input Length Generalization. ICLR.
293. [293]T. Preston-Werner (2024) Semantic Versioning 2.0.0. External Links: Link
294. [294]P. Putta, E. Mills, N. Garg, et al. (2024) Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents. arXiv Preprint arXiv:2408.07199.
295. [295]PyTorch Team (2026) Miles: A PyTorch-Native Stack for Large-Scale LLM RL Post-Training. Note: PyTorch Blog External Links: Link
296. [296]P. Qi, X. Wan, G. Huang, and M. Lin (2023) Zero Bubble Pipeline Parallelism. arXiv Preprint arXiv:2401.10241.
297. [297]Z. Qi, M. Wan, J. Cao, and M. Lin (2024) Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers. arXiv Preprint arXiv:2408.06195.
298. [298]Y. Qin, X. Li, H. Zou, et al. (2024) O1 Replication Journey: A Strategic Progress Report. arXiv Preprint arXiv:2410.18982. External Links: Link
299. [299]Y. Qin, S. Liang, Y. Ye, et al. (2024) ToolLLM: Facilitating Large Language Models to Master 16000+ Real-World APIs. In Proceedings of the 12th International Conference on Learning Representations (ICLR), External Links: Link
300. [300]A. H. Rackauckas (2024) RAG-Fusion: A New Take on Retrieval-Augmented Generation.
301. [301]A. Radford, J. Wu, R. Child, D. Luen, D. Amodei, and I. Sutskever (2019) Language Models Are Unsupervised Multitask Learners. OpenAI Blog. External Links: Link
302. [302]R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn (2023) Direct Preference Optimization: Your Language Model Is Secretly a Reward Model. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
303. [303]C. Raffel, N. Shazeer, A. Roberts, et al. (2020) Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research. External Links: Link
304. [304]S. Rajbhandari, J. Rasley, O. Rber, and Y. He (2020) ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. arXiv Preprint arXiv:1910.02054.
305. [305]S. Rajbhandari, O. Ruwase, J. Rasley, S. Smith, and Y. He (2021) ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning. arXiv Preprint arXiv:2104.07857.
306. [306]T. Rashid, M. Samvelyan, C. S. d. Witt, G. Farquhar, J. Foerster, and S. Whiteson (2018) QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. In Proceedings of the 35th International Conference on Machine Learning (ICML), External Links: Link
307. [307]E. Rescorla (2018) The Transport Layer Security (TLS) Protocol Version 1.3. External Links: Link
308. [308]H. Robbins and S. Monro (1951) A Stochastic Approximation Method. The Annals of Mathematical Statistics.
309. [309]S. Robertson and H. Zaragoza (2009) The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval.
310. [310]D. E. Rumelhart, G. E. Hinton, and R. J. Williams (1986) Learning Representations by Back-Propagating Errors. Nature. External Links: Link
311. [311]G. A. Rummery and M. Niranjan (1994) On-Line q-Learning Using Connectionist Systems.
312. [312]V. Sanh, L. Debut, J. Chaumond, and T. Wolf (2019) DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter. arXiv Preprint arXiv:1910.01108.
313. [313]K. Santhanam, O. Khattab, J. Saad-Falcon, C. Potts, and M. Zaharia (2022) ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL),
314. [314]T. Schaul, J. Quan, I. Antonoglou, and D. Silver (2016) Prioritized Experience Replay. In Proceedings of the 4th International Conference on Learning Representations (ICLR), External Links: Link
315. [315]T. Schick, J. Dwivedi-Yu, R. Dessì, et al. (2023) Toolformer: Language Models Can Teach Themselves to Use Tools. In Advances in Neural Information Processing Systems,
316. [316]J. Schrittwieser, I. Antonoglou, T. Hubert, et al. (2020) Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model. Nature. External Links: Link
317. [317]J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz (2015) Trust Region Policy Optimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML), External Links: Link
318. [318]J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel (2016) High-Dimensional Continuous Control Using Generalized Advantage Estimation. In Proceedings of the 4th International Conference on Learning Representations (ICLR), External Links: Link
319. [319]J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal Policy Optimization Algorithms. arXiv Preprint arXiv:1707.06347. External Links: Link
320. [320]R. Sennrich, B. Haddow, and A. Birch (2016) Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the ACL, External Links: Link
321. [321]A. Sergeev and M. D. Balso (2018) Horovod: Fast and Easy Distributed Deep Learning in TensorFlow. arXiv Preprint arXiv:1802.05799.
322. [322]J. Shah, G. Bikshandi, Y. Zhang, V. Thakkar, P. Ramani, and T. Dao (2024) FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision. arXiv Preprint arXiv:2407.08691. External Links: Link
323. [323]Z. Shao, P. Wang, Q. Zhu, et al. (2024) DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv Preprint arXiv:2402.03300. External Links: Link
324. [324]L. S. Shapley (1953) Stochastic Games. In Proceedings of the National Academy of Sciences, External Links: Link
325. [325]N. Shazeer, A. Mirhoseini, K. Maziarz, et al. (2017) Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. In International Conference on Learning Representations (ICLR), External Links: Link
326. [326]N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao (2023) Reflexion: Language Agents with Verbal Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
327. [327]M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro (2019) Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv Preprint arXiv:1909.08053.
328. [328]Y. Shoham and K. Leyton-Brown (2008) Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations. External Links: Link
329. [329]M. Shridhar, X. Yuan, M. Côté, Y. Bisk, A. Trischler, and M. Hausknecht (2021) ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In International Conference on Learning Representations (ICLR),
330. [330]Significant Gravitas (2023) AutoGPT: An Autonomous GPT-4 Experiment. Note: Open-source project External Links: Link
331. [331]D. Silver, A. Huang, C. J. Maddison, et al. (2016) Mastering the Game of Go with Deep Neural Networks and Tree Search. Nature. External Links: Link
332. [332]D. Silver, J. Schrittwieser, K. Simonyan, et al. (2017) Mastering the Game of Go Without Human Knowledge. Nature. External Links: Link
333. [333]J. Skalse, N. Howe, D. Krasheninnikov, and D. Krueger (2022) Defining and Characterizing Reward Hacking. In Advances in Neural Information Processing Systems, External Links: Link
334. [334]R. G. Smith (1980) The Contract Net Protocol: High-Level Communication and Control in a Distributed Problem Solver. IEEE Transactions on Computers. External Links: Link
335. [335]C. Snell, J. Lee, K. Xu, and A. Kumar (2024) Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters. arXiv Preprint arXiv:2408.03314.
336. [336]K. Sparck Jones (1972) A Statistical Interpretation of Term Specificity and Its Application in Retrieval. Journal of Documentation.
337. [337]L. R. Squire (1992) Declarative and Nondeclarative Memory: Multiple Brain Systems Supporting Learning and Memory. Journal of Cognitive Neuroscience. External Links: Link
338. [338]P. Steinberger (2026) OpenClaw: An Always-On Agent Orchestrator. Note: Open-source project External Links: Link
339. [339]N. Stiennon, L. Ouyang, J. Wu, et al. (2020) Learning to Summarize from Human Feedback. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
340. [340]J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu (2024) RoFormer: Enhanced Transformer with Rotary Position Embedding. Neurocomputing.
341. [341]J. Su, J. Cao, W. Liu, and Y. Ou (2021) Whitening Sentence Representations for Better Semantics and Faster Retrieval. arXiv Preprint arXiv:2103.15316. External Links: Link
342. [342]S. Sukhbaatar, A. Szlam, and R. Fergus (2016) Learning Multiagent Communication with Backpropagation. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
343. [343]S. Sukhbaatar, A. Szlam, J. Weston, and R. Fergus (2015) End-to-End Memory Networks. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
344. [344]T. R. Sumers, S. Yao, K. Narasimhan, and T. L. Griffiths (2024) Cognitive Architectures for Language Agents. Transactions on Machine Learning Research (TMLR). External Links: Link
345. [345]M. Sun, Z. Liu, A. Bair, and J. Z. Kolter (2024) A Simple and Effective Pruning Approach for Large Language Models. In International Conference on Learning Representations (ICLR), External Links: Link
346. [346]R. S. Sutton and A. G. Barto (2018) Reinforcement Learning: An Introduction. External Links: Link
347. [347]R. S. Sutton (1988) Learning to Predict by the Methods of Temporal Differences. Machine Learning.
348. [348]M. P. Team (2025) TorchForge: PyTorch-Native Post-Training at Scale. External Links: Link
349. [349]P. Team (2024) Torchtune: PyTorch Native Post-Training Library. External Links: Link
350. [350]Q. Team (2024) Qwen2.5: A Party of Foundation Models. arXiv Preprint arXiv:2412.15115. External Links: Link
351. [351]Q. Team (2024) QwQ: Reflect Deeply on the Boundaries of the Unknown. Qwen Blog. External Links: Link
352. [352]Q. Team (2025) Qwen3 Technical Report. arXiv Preprint arXiv:2505.09388.
353. [353]A. Templeton, T. Conerly, J. Marcus, et al. (2024) Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread. External Links: Link
354. [354]E. Todd, M. L. Li, A. S. Sharma, A. Mueller, B. C. Wallace, and D. Bau (2024) Function Vectors in Large Language Models. In Proceedings of the 12th International Conference on Learning Representations (ICLR), External Links: Link
355. [355]M. Towers, A. Kwiatkowski, J. Terry, et al. (2024) Gymnasium: A Standard Interface for Reinforcement Learning Environments. NeurIPS Datasets and Benchmarks. External Links: Link
356. [356]E. Tulving (1985) Memory and Consciousness. Canadian Psychology / Psychologie Canadienne.
357. [357]A. Vaswani, N. Shazeer, N. Parmar, et al. (2017) Attention Is All You Need. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
358. [358]Vercel (2024) Vercel AI SDK. External Links: Link
359. [359]J. Vig (2019) A Multiscale Visualization of Attention in the Transformer Model. In Proceedings of the 57th ACL: System Demonstrations, External Links: Link
360. [360]A. K. Vijayakumar, M. Cogswell, R. R. Selvaraju, et al. (2018) Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models. AAAI.
361. [361]E. Voita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov (2019) Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link
362. [362]G. Wang, Y. Xie, Y. Jiang, et al. (2023) Voyager: An Open-Ended Embodied Agent with Large Language Models. arXiv Preprint arXiv:2305.16291. External Links: Link
363. [363]J. Wang, J. Duan, Y. Liu, Z. Yue, H. Tong, and J. Wang (2024) Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints. In Proceedings of the 12th International Conference on Learning Representations (ICLR), External Links: Link
364. [364]L. Wang, W. Xu, Y. Lan, et al. (2023) Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link
365. [365]P. Wang, L. Li, Z. Shao, et al. (2024) Math-Shepherd: Verify and Reinforce LLMs Step-by-Step Without Human Annotations. arXiv Preprint arXiv:2312.08935. External Links: Link
366. [366]X. Wang, B. Ding, Z. Hoang, et al. (2024) OpenHands: An Open Platform for AI Software Developers as Generalist Agents. arXiv Preprint arXiv:2407.16741.
367. [367]X. Wang, J. Wei, D. Schuurmans, Q. Le, et al. (2023) Self-Consistency Improves Chain of Thought Reasoning in Language Models. In Proceedings of the 11th International Conference on Learning Representations (ICLR), External Links: Link
368. [368]X. Wang, J. Wei, D. Schuurmans, Q. V. Le, et al. (2023) Self-Consistency Improves Chain of Thought Reasoning in Language Models. In International Conference on Learning Representations (ICLR), External Links: Link
369. [369]Y. Wang et al. (2025) RLFactory: A Plug-and-Play Framework for Multi-Round Tool-Use RL. Note: arXiv preprint
370. [370]Y. e. a. Wang (2025) Improving Data Efficiency for LLM Reinforcement Fine-Tuning via Difficulty-Targeted Online Data Selection. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
371. [371]Y. Wang, Y. Kordi, S. Mishra, et al. (2022) Self-Instruct: Aligning Language Models with Self-Generated Instructions. arXiv Preprint arXiv:2212.10560. External Links: Link
372. [372]Z. Wang et al. (2024) Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts. Note: arXiv preprint
373. [373]C. J. C. H. Watkins (1989) Learning from Delayed Rewards.
374. [374]J. Wei, X. Wang, D. Schuurmans, et al. (2022) Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
375. [375]G. Weiss and ed (1999) Multiagent Systems: A Modern Approach to Distributed Artificial Intelligence.
376. [376]L. v. Werra, Y. Belkada, L. Tunstall, et al. (2022) TRL: Transformer Reinforcement Learning. External Links: Link
377. [377]J. Weston, S. Chopra, and A. Bordes (2015) Memory Networks. In International Conference on Learning Representations (ICLR), External Links: Link
378. [378]B. T. Willard and R. Louf (2023) Efficient Guided Generation for Large Language Models. arXiv Preprint arXiv:2307.09702. External Links: Link
379. [379]R. J. Williams (1992) Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning. External Links: Link
380. [380]E. B. Wilson (1927) Probable Inference, the Law of Succession, and Statistical Inference. Journal of the American Statistical Association. External Links: Link
381. [381]M. Wooldridge, N. R. Jennings, and D. Kinny (2000) The Gaia Methodology for Agent-Oriented Analysis and Design. Autonomous Agents and Multi-Agent Systems.
382. [382]M. Wooldridge (2009) An Introduction to MultiAgent Systems.
383. [383]B. Workshop (2023) BLOOM: A 176B-Parameter Open-Access Multilingual Language Model. arXiv Preprint arXiv:2211.05100. External Links: Link
384. [384]D. Wu, H. Wang, W. Yu, Y. Zhang, K. Chang, and D. Yu (2025) LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. In International Conference on Learning Representations (ICLR), External Links: Link
385. [385]Q. Wu, G. Bansal, J. Zhang, et al. (2023) AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. arXiv Preprint arXiv:2308.08155.
386. [386]Y. Wu et al. (2026) Long-Horizon-Terminal-Bench: Dense Reward Grading for Command-Line Agents. Note: arXiv preprint
387. [387]Z. Wu, A. Arora, Z. Wang, B. Kim, and T. Huang (2024) Retrieval Head Mechanistically Explains Long-Context Factuality. arXiv Preprint arXiv:2404.15574. External Links: Link
388. [388]Z. Wu, Q. Hu, Y. Zhang, Y. Gao, and J. Chen (2024) An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models. arXiv Preprint arXiv:2408.00724.
389. [389]xAI (2026) Grok 4.5: Training on Real Developer Session Traces. Note: Technical report
390. [390]Z. Xi, Y. Ding, W. Chen, et al. (2024) AgentGym: Evolving Large Language Model-Based Agents Across Diverse Environments. arXiv Preprint arXiv:2406.04151. External Links: Link
391. [391]F. Xia, T. Liu, J. Wang, W. Zhang, and H. Li (2008) Listwise Approach to Learning to Rank: Theory and Algorithm. In Proceedings of the 25th International Conference on Machine Learning (ICML),
392. [392]G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han (2023) SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. In Proceedings of the 40th International Conference on Machine Learning (ICML), External Links: Link
393. [393]G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis (2024) Efficient Streaming Language Models with Attention Sinks. In Proceedings of the 12th International Conference on Learning Representations (ICLR), External Links: Link
394. [394]G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis (2024) Efficient Streaming Language Models with Attention Sinks. In International Conference on Learning Representations (ICLR), External Links: Link
395. [395]S. Xiao, Z. Liu, P. Zhang, and N. Muennighoff (2023) C-Pack: Packaged Resources to Advance General Chinese Embedding. arXiv Preprint arXiv:2309.07597.
396. [396]S. M. Xie, A. Raghunathan, P. Liang, and T. Ma (2022) An Explanation of in-Context Learning as Implicit Bayesian Inference. In Proceedings of the 10th International Conference on Learning Representations (ICLR), External Links: Link
397. [397]T. Xie, D. Zhang, J. Chen, et al. (2024) OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. In Advances in Neural Information Processing Systems,
398. [398]C. Xu, Q. Sun, K. Zheng, et al. (2023) WizardLM: Empowering Large Language Models to Follow Complex Instructions. arXiv Preprint arXiv:2304.12244. External Links: Link
399. [399]H. Xu, H. Zhao, Y. Liu, and D. Wei (2025) It Takes Two: Pairwise Preference Optimization with Two Rollouts. arXiv Preprint arXiv:2505.07856. External Links: Link
400. [400]W. Xu, Z. Liang, K. Mei, H. Gao, J. Tan, and Y. Zhang (2025) A-Mem: Agentic Memory for LLM Agents. In Advances in Neural Information Processing Systems (NeurIPS),
401. [401]S. Yan, J. Gu, Y. Zhu, and Z. Ling (2024) Corrective Retrieval Augmented Generation. arXiv Preprint arXiv:2401.15884. External Links: Link
402. [402]C. Yang, X. Wang, Y. Lu, et al. (2024) Large Language Models as Optimizers. In Proceedings of the 12th International Conference on Learning Representations (ICLR), External Links: Link
403. [403]J. Yang, Y. Zhu, B. Wang, et al. (2025) ARQ: Attentive Reasoning Queries for Multi-Hop Question Answering over Long Contexts. arXiv Preprint arXiv:2501.08290. External Links: Link
404. [404]J. Yang, C. E. Jimenez, A. Wettig, et al. (2024) SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
405. [405]Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le (2019) XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Advances in Neural Information Processing Systems (NeurIPS),
406. [406]Z. Yang, P. Qi, S. Zhang, et al. (2018) HotpotQA: A Dataset for Diverse, Explainable Multi-Hop Question Answering. In Proceedings of EMNLP,
407. [407]S. Yao, D. Yu, J. Zhao, et al. (2023) Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
408. [408]S. Yao, D. Yu, J. Zhao, et al. (2024) Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
409. [409]S. Yao, J. Zhao, D. Yu, et al. (2023) ReAct: Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations (ICLR), External Links: Link
410. [410]Z. Yao, S. Rajbhandari, R. Y. Aminabadi, et al. (2023) DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-Like Models at All Scales. arXiv Preprint arXiv:2308.01320.
411. [411]J. Yoo and D. Shin (2025) Adaptive Episode Length Adjustment for Multi-Agent Reinforcement Learning. In Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS),
412. [412]G. Yu, J. S. Jeong, G. Kim, S. Kim, and B. Chun (2022) Orca: A Distributed Serving System for Transformer-Based Generative Models. In Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI),
413. [413]Q. Yu, Z. Sun, S. Wen, et al. (2025) DAPO: An Open-Source LLM Reinforcement Learning System. arXiv Preprint arXiv:2503.14476. External Links: Link
414. [414]X. Yu, B. Peng, R. Xu, et al. (2026) Reinforcement World Model Learning for LLM-Based Agents. arXiv Preprint arXiv:2602.05842.
415. [415]T. Zadouri, J. Shah, G. Bikshandi, and T. Dao (2026) FlashAttention-4: Hardware-Efficient Attention on Blackwell GPUs with Minimal Software Design. arXiv Preprint arXiv:2603.05451. External Links: Link
416. [416]M. Zaheer, G. Guruganesh, K. A. Dubey, et al. (2020) Big Bird: Transformers for Longer Sequences. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
417. [417]E. B. Zaken, S. Ravfogel, and Y. Goldberg (2022) BitFit: Simple Parameter-Efficient Fine-Tuning for Transformer-Based Masked Language-Models. In Proceedings of the 60th Annual Meeting of the ACL, External Links: Link
418. [418]E. Zelikman, G. Harik, Y. Shao, V. Jayasiri, N. Haber, and N. D. Goodman (2024) Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking. arXiv Preprint arXiv:2403.09629. External Links: Link
419. [419]E. Zelikman, Y. Wu, J. Mu, and N. D. Goodman (2022) STaR: Bootstrapping Reasoning with Reasoning. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
420. [420]A. L. Zhang, S. H. Mahdavi, P. Liang, and T. Hashimoto (2025) Recursive Language Models. arXiv Preprint arXiv:2512.24601. External Links: Link
421. [421]B. Zhang and R. Sennrich (2019) Root Mean Square Layer Normalization. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
422. [422]J. Zhang, S. Yu, D. Chong, et al. (2025) Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity. arXiv Preprint arXiv:2510.01171. External Links: Link
423. [423]K. Zhang, Z. Yang, and T. Başar (2021) Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms. Handbook of Reinforcement Learning and Control. External Links: Link
424. [424]Q. Zhang, M. Chen, A. Bukharin, et al. (2023) AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. arXiv Preprint arXiv:2303.10512. External Links: Link
425. [425]T. Zhang, S. G. Patil, N. Jain, et al. (2024) RAFT: Adapting Language Model to Domain Specific RAG. arXiv Preprint arXiv:2403.10131. External Links: Link
426. [426]T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi (2020) BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations (ICLR), External Links: Link
427. [427]W. Zhang et al. (2026) Self-Evolving Software Agents: Combining BDI Architectures with LLMs. Note: arXiv preprint
428. [428]X. Zhang, Y. Chen, S. Hu, et al. (2024) InfiniteBench: Extending Long Context Evaluation Beyond 100K Tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link
429. [429]Z. Zhang, Y. Sheng, T. Zhou, et al. (2023) H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
430. [430]S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover (2026) Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. In Proceedings of ICLR 2026, Note: arXiv preprint arXiv:2601.18734 External Links: Link
431. [431]Y. Zhao, A. Gu, R. Varma, et al. (2023) PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel. In Proceedings of the VLDB Endowment,
432. [432]Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. J. Liu (2023) SLiC-HF: Sequence Likelihood Calibration with Human Feedback. arXiv Preprint arXiv:2305.10425. External Links: Link
433. [433]L. Zheng, W. Chiang, Y. Sheng, et al. (2023) Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
434. [434]Zhipu AI (2025) GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models. arXiv preprint arXiv:2508.06471. External Links: Link
435. [435]Zhipu AI (2026) GLM-5: From Vibe Coding to Agentic Engineering. arXiv preprint arXiv:2602.15763. External Links: Link
436. [436]Y. Zhong, Y. Chen, Z. Li, and M. Chen (2025) Importance Sampling Corrections for Large Language Model Alignment with Asynchronous Generation. arXiv Preprint arXiv:2503.09057. External Links: Link
437. [437]Y. Zhong, J. Shi, Y. Chen, and M. Chen (2025) GDPO: Learning to Directly Align Language Models with Group-Decoupled Reward. arXiv Preprint arXiv:2501.17888. External Links: Link
438. [438]A. Zhou, K. Yan, M. Shlapentokh-Rothman, H. Wang, and Y. Wang (2024) Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models. arXiv Preprint arXiv:2310.04406.
439. [439]C. Zhou, P. Liu, P. Xu, et al. (2023) LIMA: Less Is More for Alignment. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
440. [440]S. Zhou, F. F. Xu, H. Zhu, et al. (2024) WebArena: A Realistic Web Environment for Building Autonomous Agents. In International Conference on Learning Representations (ICLR), External Links: Link
441. [441]Y. Zhou, A. I. Muresanu, Z. Han, et al. (2023) Large Language Models Are Human-Level Prompt Engineers. In Proceedings of the 11th International Conference on Learning Representations (ICLR), External Links: Link
442. [442]D. M. Ziegler, N. Stiennon, J. Wu, et al. (2019) Fine-Tuning Language Models from Human Preferences. arXiv Preprint arXiv:1909.08593. External Links: Link
443. [443]A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson (2023) Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv Preprint arXiv:2307.15043. External Links: Link