A Functionality-Grounded Benchmark for Evaluating Web Agents in E-commerce Domains

TL;DR

This paper introduces Amazon-Bench, a comprehensive benchmark for evaluating web agents in e-commerce, focusing on functionality coverage and safety, revealing current agents' limitations in complex tasks and risk management.

Contribution

The paper presents Amazon-Bench, a new benchmark with a data generation pipeline and evaluation framework for assessing functionality and safety of web agents in e-commerce.

Findings

Current agents struggle with complex, multi-step queries.

Existing agents pose safety risks like incorrect purchases or account changes.

Evaluation reveals the need for more robust and reliable web agents.

Abstract

Web agents have shown great promise in performing many tasks on ecommerce website. To assess their capabilities, several benchmarks have been introduced. However, current benchmarks in the e-commerce domain face two major problems. First, they primarily focus on product search tasks (e.g., Find an Apple Watch), failing to capture the broader range of functionalities offered by real-world e-commerce platforms such as Amazon, including account management and gift card operations. Second, existing benchmarks typically evaluate whether the agent completes the user query, but ignore the potential risks involved. In practice, web agents can make unintended changes that negatively impact the user account or status. For instance, an agent might purchase the wrong item, delete a saved address, or incorrectly configure an auto-reload setting. To address these gaps, we propose a new benchmark called Amazon-Bench. To generate user queries that cover a broad range of tasks, we propose a data generation pipeline that leverages webpage content and interactive elements (e.g., buttons, check boxes) to create diverse, functionality-grounded user queries covering tasks such as address management, wish list management, and brand store following. To improve the agent evaluation, we propose an automated evaluation framework that assesses both the performance and the safety of web agents. We systematically evaluate different agents, finding that current agents struggle with complex queries and pose safety risks. These results highlight the need for developing more robust and reliable web agents.