Large Language Models Can Self-Improve At Web Agent Tasks

TL;DR

This paper demonstrates that large language models can significantly improve their web navigation and task completion abilities through self-generated fine-tuning, achieving notable performance gains on the WebArena benchmark.

Contribution

It introduces a self-improvement method for LLM-based web agents, showing a 31% performance boost and developing new evaluation metrics for detailed assessment.

Findings

31% improvement in task completion rate

Effective self-fine-tuning on synthetic data

Novel metrics for evaluating agent trajectories

Abstract

Training models to act as agents that can effectively navigate and perform actions in a complex environment, such as a web browser, has typically been challenging due to lack of training data. Large language models (LLMs) have recently demonstrated some capability to navigate novel environments as agents in a zero-shot or few-shot fashion, purely guided by natural language instructions as prompts. Recent research has also demonstrated LLMs have the capability to exceed their base performance through self-improvement, i.e. fine-tuning on data generated by the model itself. In this work, we explore the extent to which LLMs can self-improve their performance as agents in long-horizon tasks in a complex environment using the WebArena benchmark. In WebArena, an agent must autonomously navigate and perform actions on web pages to achieve a specified objective. We explore fine-tuning on three distinct synthetic training data mixtures and achieve a 31\% improvement in task completion rate over the base model on the WebArena benchmark through a self-improvement procedure. We additionally contribute novel evaluation metrics for assessing the performance, robustness, capabilities, and quality of trajectories of our fine-tuned agent models to a greater degree than simple, aggregate-level benchmark scores currently used to measure self-improvement.