WebSight: A Vision-First Architecture for Robust Web Agents

TL;DR

WebSight introduces a vision-only web agent architecture with a specialized vision-language model, achieving high accuracy and success rates in web navigation tasks without relying on HTML or DOM inputs.

Contribution

The paper presents WebSight, a novel vision-first architecture and a new model, WebSight-7B, optimized for visual web interaction, outperforming existing models on key benchmarks.

Findings

WebSight-7B achieves 58.84% accuracy on Showdown Clicks.

WebSight reaches 68.0% success on WebVoyager benchmark.

High task accuracy of 97.14% indicates precise web navigation.

Abstract

We introduce WebSight, a vision-based autonomous web agent, designed to interact with web environments purely through visual perception, eliminating dependence on HTML or DOM-based inputs. Central to our approach we introduce our new model, WebSight-7B, a fine-tuned vision-language model optimized for UI element interaction, trained using LoRA on a web-focused subset of the Wave-UI-25K dataset. WebSight integrates this model into a modular multi-agent architecture, comprising planning, reasoning, vision-action, and verification agents, coordinated through an episodic memory mechanism. WebSight-7B achieves a top-1 accuracy of 58.84% on the Showdown Clicks benchmark, outperforming several larger generalist models while maintaining lower latency. The full WebSight agent achieves a 68.0% success rate on the WebVoyager benchmark, surpassing systems from labs such as OpenAI (61.0%) and HCompany (Runner H, 67.0%). Among tasks completed, WebSight answers correctly 97.14% of the time, indicating high precision. Together, WebSight and WebSight-7B establish a new standard for interpretable, robust, and efficient visual web navigation.