Multimodal Auto Validation For Self-Refinement in Web Agents

TL;DR

This paper proposes a multi-modal validation and self-refinement approach for web agents, significantly improving their task completion rates by enabling automatic detection and correction of workflow failures, thus enhancing reliability in complex tasks.

Contribution

It introduces a novel auto-validator leveraging multiple modalities and hierarchy for web agent validation, along with a self-refinement mechanism for performance enhancement.

Findings

Boosted task completion rate from 76.2% to 81.24%.

Demonstrated effectiveness of multi-modal validation.

Enhanced reliability of web agents in complex scenarios.

Abstract

As our world digitizes, web agents that can automate complex and monotonous tasks are becoming essential in streamlining workflows. This paper introduces an approach to improving web agent performance through multi-modal validation and self-refinement. We present a comprehensive study of different modalities (text, vision) and the effect of hierarchy for the automatic validation of web agents, building upon the state-of-the-art Agent-E web automation framework. We also introduce a self-refinement mechanism for web automation, using the developed auto-validator, that enables web agents to detect and self-correct workflow failures. Our results show significant gains on Agent-E's (a SOTA web agent) prior state-of-art performance, boosting task-completion rates from 76.2\% to 81.24\% on the subset of the WebVoyager benchmark. The approach presented in this paper paves the way for more reliable digital assistants in complex, real-world scenarios.