Tur[k]ingBench: A Challenge Benchmark for Web Agents

TL;DR

TurkingBench is a comprehensive benchmark with diverse web-based tasks designed to evaluate multi-modal models' ability to perform complex annotation tasks on natural HTML pages, aiming to advance web agent development.

Contribution

This paper introduces TurkingBench, a novel benchmark with real HTML tasks and a framework linking chatbot responses to web actions, enabling evaluation of multi-modal web agents.

Findings

Models outperform random chance but still have significant room for improvement.

Benchmark includes 32.2K instances across 158 tasks, providing a diverse evaluation platform.

Evaluation of models like GPT-4 and InternVL demonstrates current capabilities and gaps.

Abstract

Can advanced multi-modal models effectively tackle complex web-based tasks? Such tasks are often found on crowdsourcing platforms, where crowdworkers engage in challenging micro-tasks within web-based environments. Building on this idea, we present TurkingBench, a benchmark consisting of tasks presented as web pages with textual instructions and multi-modal contexts. Unlike previous approaches that rely on artificially synthesized web pages, our benchmark uses natural HTML pages originally designed for crowdsourcing workers to perform various annotation tasks. Each task's HTML instructions are instantiated with different values derived from crowdsourcing tasks, creating diverse instances. This benchmark includes 32.2K instances spread across 158 tasks. To support the evaluation of TurkingBench, we have developed a framework that links chatbot responses to actions on web pages (e.g., modifying a text box, selecting a radio button). We assess the performance of cutting-edge private and open-source models, including language-only and vision-language models (such as GPT4 and InternVL), on this benchmark. Our results show that while these models outperform random chance, there is still significant room for improvement. We hope that this benchmark will drive progress in the evaluation and development of web-based agents.