WebCoT: Enhancing Web Agent Reasoning by Reconstructing Chain-of-Thought in Reflection, Branching, and Rollback

TL;DR

This paper introduces WebCoT, a method to improve web agent reasoning by reconstructing and fine-tuning chain-of-thought rationales for better performance in dynamic web environments.

Contribution

It identifies key reasoning skills for web agents, curates trajectory data, and demonstrates how reconstructing reasoning algorithms enhances LLM performance.

Findings

Significant performance improvements on multiple web benchmarks.

Effective distillation of reasoning patterns into LLMs through fine-tuning.

Enhanced reasoning skills lead to more robust web agent behavior.

Abstract

Web agents powered by Large Language Models (LLMs) show promise for next-generation AI, but their limited reasoning in uncertain, dynamic web environments hinders robust deployment. In this paper, we identify key reasoning skills essential for effective web agents, i.e., reflection & lookahead, branching, and rollback, and curate trajectory data that exemplifies these abilities by reconstructing the agent's (inference-time) reasoning algorithms into chain-of-thought rationales. We conduct experiments in the agent self-improving benchmark, OpenWebVoyager, and demonstrate that distilling salient reasoning patterns into the backbone LLM via simple fine-tuning can substantially enhance its performance. Our approach yields significant improvements across multiple benchmarks, including WebVoyager, Mind2web-live, and SimpleQA (web search), highlighting the potential of targeted reasoning skill enhancement for web agents.