AutoForge: Automated Environment Synthesis for Agentic Reinforcement Learning

TL;DR

AutoForge introduces an automated, scalable pipeline for synthesizing challenging simulated environments and an environment-level RL algorithm that enhances training stability and efficiency for agentic reinforcement learning.

Contribution

It presents a unified pipeline for automated environment synthesis and an environment-level RL algorithm addressing user instability and improving training performance.

Findings

Effective environment synthesis for high-difficulty tasks

Improved training stability and efficiency in agentic RL

Strong out-of-domain generalization demonstrated

Abstract

Conducting reinforcement learning (RL) in simulated environments offers a cost-effective and highly scalable way to enhance language-based agents. However, previous work has been limited to semi-automated environment synthesis or tasks lacking sufficient difficulty, offering little breadth or depth. In addition, the instability of simulated users integrated into these environments, along with the heterogeneity across simulated environments, poses further challenges for agentic RL. In this work, we propose: (1) a unified pipeline for automated and scalable synthesis of simulated environments associated with high-difficulty but easily verifiable tasks; and (2) an environment level RL algorithm that not only effectively mitigates user instability but also performs advantage estimation at the environment level, thereby improving training efficiency and stability. Comprehensive evaluations on agentic benchmarks, including tau-bench, tau2-Bench, and VitaBench, validate the effectiveness of our proposed method. Further in-depth analyses underscore its out-of-domain generalization.