Retrieval-Augmented LLM Agents: Learning to Learn from Experience

TL;DR

This paper introduces a combined training approach for retrieval-augmented LLM agents that enhances their ability to generalize to new tasks by effectively leveraging retrieved experience during in-context learning.

Contribution

It proposes a novel pipeline integrating experience retrieval with supervised fine-tuning, improving agent generalization and establishing optimal retrieval strategies.

Findings

Supervised fine-tuning with LoRA outperforms existing agent training methods.

Optimal experience retrieval strategies significantly boost generalization.

The combined approach enhances learning from experience for unseen tasks.

Abstract

While large language models (LLMs) have advanced the development of general-purpose agents, achieving robust generalization to unseen tasks remains a significant challenge. Current approaches typically rely on either fine-tuning or training-free memory-augmented generation using retrieved experience; yet both have limitations: fine-tuning often fails to extrapolate to new tasks, while experience retrieval often underperforms compared to supervised baselines. In this work, we propose to combine these approaches and systematically study how to train retrieval-augmented LLM agents to effectively leverage retrieved trajectories in-context. First, we establish a robust supervised fine-tuning (SFT) recipe using LoRA that outperforms several state-of-the-art agent training pipelines. Second, we provide a detailed analysis of key design choices for experience retrieval, identifying optimal strategies for storage, querying, and trajectory selection. Finally, we propose a pipeline that integrates experience retrieval into the fine-tuning process. Our results demonstrate that this combined approach significantly improves generalization to unseen tasks, providing a scalable and effective framework for building agents that learn to learn from experience.