Contextual Experience Replay for Self-Improvement of Language Agents

TL;DR

This paper introduces Contextual Experience Replay (CER), a training-free framework that enables language agents to self-improve by leveraging past experiences, significantly enhancing their performance in complex web navigation tasks.

Contribution

The paper proposes CER, a novel method for language agents to self-improve during inference by synthesizing past experiences into a dynamic memory buffer.

Findings

CER achieves 31.9% on VisualWebArena.

CER improves success rate by 51.0% over baseline.

CER demonstrates efficiency and validity through comprehensive analysis.

Abstract

Large language model (LLM) agents have been applied to sequential decision-making tasks such as web navigation, but without any environment-specific experiences, they often fail in these complex tasks. Moreover, current LLM agents are not designed to continually learn from past experiences during inference time, which could be crucial for them to gain these environment-specific experiences. To address this, we propose Contextual Experience Replay (CER), a training-free framework to enable efficient self-improvement for language agents in their context window. Specifically, CER accumulates and synthesizes past experiences into a dynamic memory buffer. These experiences encompass environment dynamics and common decision-making patterns, allowing the agents to retrieve and augment themselves with relevant knowledge in new tasks, enhancing their adaptability in complex environments. We evaluate CER on the challenging WebArena and VisualWebArena benchmarks. On VisualWebArena, CER achieves a competitive performance of 31.9%. On WebArena, CER also gets a competitive average success rate of 36.7%, relatively improving the success rate of the GPT-4o agent baseline by 51.0%. We also conduct a comprehensive analysis on it to prove its efficiency, validity and understand it better.