Dual-View Visual Contextualization for Web Navigation

TL;DR

This paper introduces a dual-view approach combining HTML and visual webpage features to improve web navigation agents, demonstrating superior performance on diverse real-world website tasks.

Contribution

It proposes a novel dual-view contextualization method that integrates visual and textual features of webpage elements to enhance web navigation performance.

Findings

Outperforms baseline methods across all scenarios

Effective in cross-task, cross-website, and cross-domain settings

Improves contextual understanding of webpage elements

Abstract

Automatic web navigation aims to build a web agent that can follow language instructions to execute complex and diverse tasks on real-world websites. Existing work primarily takes HTML documents as input, which define the contents and action spaces (i.e., actionable elements and operations) of webpages. Nevertheless, HTML documents may not provide a clear task-related context for each element, making it hard to select the right (sequence of) actions. In this paper, we propose to contextualize HTML elements through their "dual views" in webpage screenshots: each HTML element has its corresponding bounding box and visual content in the screenshot. We build upon the insight -- web developers tend to arrange task-related elements nearby on webpages to enhance user experiences -- and propose to contextualize each element with its neighbor elements, using both textual and visual features. The resulting representations of HTML elements are more informative for the agent to take action. We validate our method on the recently released Mind2Web dataset, which features diverse navigation domains and tasks on real-world websites. Our method consistently outperforms the baseline in all the scenarios, including cross-task, cross-website, and cross-domain ones.