Agent Lightning: Train ANY AI Agents with Reinforcement Learning

TL;DR

Agent Lightning is a versatile framework that enables reinforcement learning-based training of large language models for any AI agent, supporting diverse agent architectures with minimal modifications.

Contribution

It introduces a decoupled RL training approach, a hierarchical RL algorithm LightningRL, and a standardized interface for training and deploying diverse AI agents.

Findings

Stable improvements across multiple tasks

Supports complex multi-agent interactions

Minimal code modifications required for integration

Abstract

We present Agent Lightning, a flexible and extensible framework that enables Reinforcement Learning (RL)-based training of Large Language Models (LLMs) for any AI agent. Unlike existing methods that tightly couple RL training with agent or rely on sequence concatenation with masking, Agent Lightning achieves complete decoupling between agent execution and training, allowing seamless integration with existing agents developed via diverse ways (e.g., using frameworks like LangChain, OpenAI Agents SDK, AutoGen, and building from scratch) with almost ZERO code modifications. By formulating agent execution as Markov decision process, we define an unified data interface and propose a hierarchical RL algorithm, LightningRL, which contains a credit assignment module, allowing us to decompose trajectories generated by ANY agents into training transition. This enables RL to handle complex interaction logic, such as multi-agent scenarios and dynamic workflows. For the system design, we introduce a Training-Agent Disaggregation architecture, and brings agent observability frameworks into agent runtime, providing a standardized agent finetuning interface. Experiments across text-to-SQL, retrieval-augmented generation, and math tool-use tasks demonstrate stable, continuous improvements, showcasing the framework's potential for real-world agent training and deployment.