Rethinking Experience Utilization in Self-Evolving Language Model Agents

TL;DR

This paper investigates how self-evolving language model agents can better utilize stored experience during decision-making, demonstrating that selective, context-aware experience use improves performance across various frameworks and environments.

Contribution

It introduces ExpWeaver, a lightweight method that enables agents to selectively incorporate experience during reasoning, improving decision quality without changing experience construction.

Findings

ExpWeaver outperforms rigid experience usage strategies across multiple frameworks and environments.

Training amplifies the benefits of selective experience utilization.

Selective experience invocation correlates with higher reasoning uncertainty and decision benefit.

Abstract

Self-evolving agents improve by accumulating and reusing experience from past interactions. Existing work has largely focused on how experience is constructed, represented, and updated, while paying less attention to how experience should be used during runtime decision-making. As a result, most agents rely on rigid usage strategies, either injecting experience once at initialization or at every step, without considering whether it is needed for the current decision. This paper studies experience utilization as a critical design dimension of self-evolving agents. We ask whether agents benefit from interweaving experience use with decision-making, so that experience is invoked only when additional guidance is needed. To examine this question, we introduce {ExpWeaver}, a lightweight instantiation that leaves experience construction unchanged and modifies only runtime utilization by exposing experience as an optional resource during reasoning. Across four representative frameworks, seven LLM backbones, and three types of environments, ExpWeaver consistently achieves the best performance among different utilization strategies. Reinforcement learning experiments further show that this behavior can be amplified through training. Usage-pattern, causal ablation, and entropy-based analyses reveal that ExpWeaver enables agents to invoke experience selectively, at beneficial decision points, and under higher reasoning uncertainty. Overall, our findings call for a shift from merely studying \emph{what} experience to store toward understanding \emph{how} and \emph{when} experience should enter decision-making.