Unveiling Disparities in Web Task Handling Between Human and Web Agent

TL;DR

This paper compares human and web agent performance in web tasks, revealing key differences in planning, reflection, and knowledge updating, which inform future agent design improvements.

Contribution

It provides a detailed analysis of human versus web agent behaviors and highlights areas for enhancing agent planning and reflection capabilities.

Findings

Humans explore and modify plans based on new information.

Differences in knowledge updating between humans and agents.

Humans better handle ambiguity and investigate failure reasons.

Abstract

With the advancement of Large-Language Models (LLMs) and Large Vision-Language Models (LVMs), agents have shown significant capabilities in various tasks, such as data analysis, gaming, or code generation. Recently, there has been a surge in research on web agents, capable of performing tasks within the web environment. However, the web poses unforeseeable scenarios, challenging the generalizability of these agents. This study investigates the disparities between human and web agents' performance in web tasks (e.g., information search) by concentrating on planning, action, and reflection aspects during task execution. We conducted a web task study with a think-aloud protocol, revealing distinct cognitive actions and operations on websites employed by humans. Comparative examination of existing agent structures and human behavior with thought processes highlighted differences in knowledge updating and ambiguity handling when performing the task. Humans demonstrated a propensity for exploring and modifying plans based on additional information and investigating reasons for failure. These findings offer insights into designing planning, reflection, and information discovery modules for web agents and designing the capturing method for implicit human knowledge in a web task.