CuES: A Curiosity-driven and Environment-grounded Synthesis Framework for Agentic RL

TL;DR

CuES introduces a curiosity-driven framework that autonomously generates meaningful tasks from environment structure, enabling scalable agent learning without predefined tasks, and improving policy performance across diverse environments.

Contribution

The paper presents CuES, a novel environment-grounded task synthesis method that eliminates the need for handcrafted tasks in agentic RL by leveraging curiosity and environment structure.

Findings

CuES produces diverse, executable task distributions matching or surpassing curated datasets.

CuES enhances downstream policy performance significantly.

The framework is effective across multiple complex environments.

Abstract

Large language model based agents are increasingly deployed in complex, tool augmented environments. While reinforcement learning provides a principled mechanism for such agents to improve through interaction, its effectiveness critically depends on the availability of structured training tasks. In many realistic settings, however, no such tasks exist a challenge we term task scarcity, which has become a key bottleneck for scaling agentic RL. Existing approaches typically assume predefined task collections, an assumption that fails in novel environments where tool semantics and affordances are initially unknown. To address this limitation, we formalize the problem of Task Generation for Agentic RL, where an agent must learn within a given environment that lacks predefined tasks. We propose CuES, a Curiosity driven and Environment grounded Synthesis framework that autonomously generates diverse, executable, and meaningful tasks directly from the environment structure and affordances, without relying on handcrafted seeds or external corpora. CuES drives exploration through intrinsic curiosity, abstracts interaction patterns into reusable task schemas, and refines them through lightweight top down guidance and memory based quality control. Across three representative environments, AppWorld, BFCL, and WebShop, CuES produces task distributions that match or surpass manually curated datasets in both diversity and executability, yielding substantial downstream policy improvements. These results demonstrate that curiosity driven, environment grounded task generation provides a scalable foundation for agents that not only learn how to act, but also learn what to learn. The code is available at https://github.com/modelscope/AgentEvolver/tree/main/research/CuES.