Comparing Human Oversight Strategies for Computer-Use Agents

TL;DR

This study compares four oversight strategies for AI agents, revealing that oversight effectiveness depends on how supervision surfaces critical moments and influences user trust and intervention success.

Contribution

It introduces a structural coordination framework for oversight strategies and empirically evaluates their impact on user interaction and trust in live web environments.

Findings

Oversight strategy influences exposure to problematic actions more than correction ability.

Plan-based strategies reduce problematic actions but do not improve intervention success.

No single oversight strategy is best across all subjective measures, with trust varying by context.

Abstract

LLM-powered computer-use agents (CUAs) are shifting users from direct manipulation to supervisory coordination. Existing oversight mechanisms, however, have largely been studied as isolated interface features, making broader oversight strategies difficult to compare. We conceptualize CUA oversight as a structural coordination problem defined by delegation structure and engagement level, and use this lens to compare four oversight strategies in a mixed-methods study with 48 participants in a live web environment. Our results show that oversight strategy more reliably shaped users' exposure to problematic actions than their ability to correct them once visible. Plan-based strategies were associated with lower rates of agent problematic-action occurrence, but not equally strong gains in runtime intervention success once such actions became visible. On subjective measures, no single strategy was uniformly best, and the clearest context-sensitive differences appeared in trust. Qualitative findings further suggest that intervention depended not only on what controls users retained, but on whether risky moments became legible as requiring judgment during execution. These findings suggest that effective CUA oversight is not achieved by maximizing human involvement alone. Instead, it depends on how supervision is structured to surface decision-critical moments and support their recognition in time for meaningful intervention.