The World Won't Stay Still: Programmable Evolution for Agent Benchmarks

TL;DR

This paper introduces ProEvolve, a graph-based framework for creating evolving environment benchmarks to evaluate the adaptability of tool-calling agents in dynamic settings.

Contribution

ProEvolve provides a programmable, graph-based method for modeling and generating evolving environments, enabling better assessment of agent robustness over time.

Findings

ProEvolve successfully generates dynamic environments in e-commerce and airline booking domains.

The framework allows explicit modeling of environment changes through graph transformations.

Agents' performance varies with environment evolution, highlighting the importance of adaptability.

Abstract

LLM-powered tool-calling agents fulfill user requests by interacting with environments, querying data, and invoking tools in a multi-turn process. Yet, most existing benchmarks evaluate these systems under static environment interfaces, with fixed schemas and toolsets, making it difficult to assess how agents behave as environments evolves -- when capabilities are added, reorganized, or deprecated across successive environment versions. In this paper, we study structured environment evolution as a benchmark-construction problem for tool-calling agents. We propose ProEvolve, a graph-based framework that makes environment evolution programmable. At its core, a typed relational graph provides a unified, explicit representation of the environment - data, tools, and schema. Under this formalism, adding, removing, or modifying capabilities are expressed as graph transformations that coherently propagate updates across tools, schemas, and data access. Building on this, ProEvolve supports (1) automatic generation of evolved executable environments through explicit graph transformations, and (2) graph-grounded construction of task sandboxes via subgraph sampling and instantiation. We validate ProEvolve in two tool-calling domains, e-commerce and airline booking, in terms of quality, implementation validity, and failure modes. Finally, we use the generated benchmark as a downstream diagnostic to study how representative agents behave under structured environment evolution.