Training Language Agents to Learn from Experience

TL;DR

This paper introduces a framework and training pipeline for language agents to learn from experience and improve performance on unseen tasks through self-reflection and reinforcement learning.

Contribution

It presents the In-context Training (ICT) framework and a reinforcement learning method for training reflectors that enable language agents to learn from experience without human examples.

Findings

Trained reflectors outperform untrained baselines on most task families.

Agents generalize to different environments beyond the training benchmark.

MetaGym library facilitates future research on self-improving language agents.

Abstract

Language agents can adapt from experience in interactive environments, but current reflection-based methods can only self-correct within a single task instance. Whether such experience can be distilled into reusable lessons that improve performance on future unseen tasks remains unclear. We address this problem by introducing the In-context Training (ICT) task, a framework for evaluating cross-task self-improvement in language agents. In ICT, a reflector model observes trajectories collected by an actor model and generates system prompts intended to improve the actor's performance on future unseen tasks. We then propose an RL-based training pipeline for learning such reflections directly from experience, without human-provided examples. Across ALFWorld and MiniHack, our trained reflectors outperform an untrained baseline on most held-out task families, showing that the ability to learn from experience can itself be learned. In some cases, we observe generalisation beyond the benchmark on which the reflector was trained, to substantially different environments. Finally, we introduce MetaGym, a generic Python library for constructing meta-environments, enabling future research on self-improving language agents.