Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning

TL;DR

This paper introduces Agent World Model (AWM), a synthetic environment generation pipeline that creates diverse, reliable, and code-driven environments for training reinforcement learning agents, enabling better generalization and more efficient interactions.

Contribution

The paper presents AWM, a scalable pipeline for generating synthetic environments with rich toolsets, improving over LLM-based environments in reliability and efficiency for agent training.

Findings

Training in synthetic environments improves out-of-distribution generalization.

AWM enables large-scale reinforcement learning with reliable reward functions.

Synthetic environments outperform benchmark-specific environments in certain tasks.

Abstract

Recent advances in large language model (LLM) have empowered autonomous agents to perform complex tasks that require multi-turn interactions with tools and environments. However, scaling such agent training is limited by the lack of diverse and reliable environments. In this paper, we propose Agent World Model (AWM), a fully synthetic environment generation pipeline. Using this pipeline, we scale to 1,000 environments covering everyday scenarios, in which agents can interact with rich toolsets (35 tools per environment on average) and obtain high-quality observations. Notably, these environments are code-driven and backed by databases, providing more reliable and consistent state transitions than environments simulated by LLMs. Moreover, they enable more efficient agent interaction compared with collecting trajectories from realistic environments. To demonstrate the effectiveness of this resource, we perform large-scale reinforcement learning for multi-turn tool-use agents. Thanks to the fully executable environments and accessible database states, we can also design reliable reward functions. Experiments on three benchmarks show that training exclusively in synthetic environments, rather than benchmark-specific ones, yields strong out-of-distribution generalization. The code is available at https://github.com/Snowflake-Labs/agent-world-model.