AutoRefine: From Trajectories to Reusable Expertise for Continual LLM Agent Refinement

TL;DR

AutoRefine is a framework that enhances continual learning in large language model agents by extracting, maintaining, and refining reusable experience patterns from execution histories, improving task performance and efficiency.

Contribution

It introduces dual-form experience pattern extraction and a maintenance mechanism to prevent repository degradation, advancing continual learning in LLM agents.

Findings

Achieved high success rates on multiple benchmarks (up to 98.4%).

Reduced steps by 20-73% in task execution.

Automatic pattern extraction outperforms manual methods.

Abstract

Large language model agents often fail to accumulate knowledge from experience, treating each task as an independent challenge. Recent methods extract experience as flattened textual knowledge, which cannot capture procedural logic of complex subtasks. They also lack maintenance mechanisms, causing repository degradation as experience accumulates. We introduce AutoRefine, a framework that extracts and maintains dual-form Experience Patterns from agent execution histories. For procedural subtasks, we extract specialized subagents with independent reasoning and memory. For static knowledge, we extract skill patterns as guidelines or code snippets. A continuous maintenance mechanism scores, prunes, and merges patterns to prevent repository degradation. Evaluated on ALFWorld, ScienceWorld, and TravelPlanner, AutoRefine achieves 98.4%, 70.4%, and 27.1% respectively, with 20-73% step reductions. On TravelPlanner, automatic extraction exceeds manually designed systems (27.1% vs 12.1%), demonstrating its ability to capture procedural coordination.