AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?

TL;DR

This paper introduces AssistantBench, a new benchmark for evaluating web agents on realistic, time-consuming tasks, revealing current limitations and proposing a new agent, SeePlanAct, that outperforms previous models.

Contribution

The paper presents AssistantBench, a comprehensive benchmark for web agents, and introduces SeePlanAct, a novel web agent that significantly improves performance on complex tasks.

Findings

Current models score below 26 on AssistantBench.

Closed-book LMs have low precision and hallucinate facts.

State-of-the-art web agents perform near zero on the benchmark.

Abstract

Language agents, built on top of language models (LMs), are systems that can interact with complex environments, such as the open web. In this work, we examine whether such agents can perform realistic and time-consuming tasks on the web, e.g., monitoring real-estate markets or locating relevant nearby businesses. We introduce AssistantBench, a challenging new benchmark consisting of 214 realistic tasks that can be automatically evaluated, covering different scenarios and domains. We find that AssistantBench exposes the limitations of current systems, including language models and retrieval-augmented language models, as no model reaches an accuracy of more than 26 points. While closed-book LMs perform well in terms of accuracy, they exhibit low precision and tend to hallucinate facts. State-of-the-art web agents reach a score of near zero. Additionally, we introduce SeePlanAct (SPA), a new web agent that significantly outperforms previous agents, and an ensemble of SPA and closed-book models reaches the best overall performance. Moreover, we analyze failures of current systems and highlight that open web navigation remains a major challenge.