WebQuest: A Benchmark for Multimodal QA on Web Page Sequences

TL;DR

WebQuest introduces a challenging multimodal web page sequence dataset for multi-page question answering, highlighting the need for advanced reasoning across multiple web pages and evaluating current models' capabilities.

Contribution

The paper presents WebQuest, a novel benchmark for multimodal multi-page QA that emphasizes real-world web reasoning and evaluates both proprietary and open-source models.

Findings

Significant performance gap between single-screen and multi-screen reasoning.

Evaluation of leading models reveals limitations in multi-page reasoning.

Chain-of-Thought prompting improves multi-screen inference capabilities.

Abstract

The rise of powerful multimodal LLMs has enhanced the viability of building web agents which can, with increasing levels of autonomy, assist users to retrieve information and complete tasks on various human-computer interfaces. It is hence necessary to build challenging benchmarks that span a wide-variety of use cases reflecting real-world usage. In this work, we present WebQuest, a multi-page question-answering dataset that requires reasoning across multiple related web pages. In contrast to existing UI benchmarks that focus on multi-step web navigation and task completion, our dataset evaluates information extraction, multimodal retrieval and composition of information from many web pages. WebQuest includes three question categories: single-screen QA, multi-screen QA, and QA based on navigation traces. We evaluate leading proprietary multimodal models like GPT-4V, Gemini Flash, Claude 3, and open source models like InstructBLIP, PaliGemma on our dataset, revealing a significant gap between single-screen and multi-screen reasoning. Finally, we investigate inference time techniques like Chain-of-Thought prompting to improve model capabilities on multi-screen reasoning.