WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Tasks

TL;DR

WebSP-Eval introduces a new framework for evaluating web agents on website security and privacy tasks, highlighting current models' limitations in handling UI elements like toggles and checkboxes.

Contribution

The paper presents a comprehensive evaluation framework, including a dataset, agent management system, and automated evaluator, for assessing web agents on security and privacy tasks.

Findings

Current models have limited exploration capabilities for security tasks.

Models struggle with specific websites and task categories.

UI elements like toggles and checkboxes cause over 45% failure rate.

Abstract

Web agents automate browser tasks, ranging from simple form completion to complex workflows like ordering groceries. While current benchmarks evaluate general-purpose performance~(e.g., WebArena) or safety against malicious actions~(e.g., SafeArena), no existing framework assesses an agent's ability to successfully execute user-facing website security and privacy tasks, such as managing cookie preferences, configuring privacy-sensitive account settings, or revoking inactive sessions. To address this gap, we introduce WebSP-Eval, an evaluation framework for measuring web agent performance on website security and privacy tasks. WebSP-Eval comprises 1) a manually crafted task dataset of 200 task instances across 28 websites; 2) a robust agentic system supporting account and initial state management across runs using a custom Google Chrome extension; and 3) an automated evaluator. We evaluate a total of 8 web agent instantiations using state-of-the-art multimodal large language models, conducting a fine-grained analysis across websites, task categories, and UI elements. Our evaluation reveals that current models suffer from limited autonomous exploration capabilities to reliably solve website security and privacy tasks, and struggle with specific task categories and websites. Crucially, we identify stateful UI elements such as toggles and checkboxes are a primary reason for agent failure, failing at a rate of more than 45\% in tasks containing these elements across many models.