WebWalker: Benchmarking LLMs in Web Traversal

TL;DR

WebWalkerQA is a benchmark for evaluating large language models' ability to perform web traversal, using a multi-agent framework that mimics human navigation to improve data extraction for complex tasks.

Contribution

The paper introduces WebWalker, a novel multi-agent framework for web traversal, and WebWalkerQA, a benchmark to assess LLMs' web navigation capabilities, addressing limitations of traditional search engines.

Findings

WebWalkerQA is a challenging benchmark for LLM web traversal.

WebWalker combined with RAG improves data retrieval in complex scenarios.

The framework demonstrates effectiveness in real-world web navigation tasks.

Abstract

Retrieval-augmented generation (RAG) demonstrates remarkable performance across tasks in open-domain question-answering. However, traditional search engines may retrieve shallow content, limiting the ability of LLMs to handle complex, multi-layered information. To address it, we introduce WebWalkerQA, a benchmark designed to assess the ability of LLMs to perform web traversal. It evaluates the capacity of LLMs to traverse a website's subpages to extract high-quality data systematically. We propose WebWalker, which is a multi-agent framework that mimics human-like web navigation through an explore-critic paradigm. Extensive experimental results show that WebWalkerQA is challenging and demonstrates the effectiveness of RAG combined with WebWalker, through the horizontal and vertical integration in real-world scenarios.