WebSight: A Vision-First Architecture for Robust Web Agents
Tanvir Bhathal, Asanshay Gupta

TL;DR
WebSight introduces a vision-only web agent architecture with a specialized vision-language model, achieving high accuracy and success rates in web navigation tasks without relying on HTML or DOM inputs.
Contribution
The paper presents WebSight, a novel vision-first architecture and a new model, WebSight-7B, optimized for visual web interaction, outperforming existing models on key benchmarks.
Findings
WebSight-7B achieves 58.84% accuracy on Showdown Clicks.
WebSight reaches 68.0% success on WebVoyager benchmark.
High task accuracy of 97.14% indicates precise web navigation.
Abstract
We introduce WebSight, a vision-based autonomous web agent, designed to interact with web environments purely through visual perception, eliminating dependence on HTML or DOM-based inputs. Central to our approach we introduce our new model, WebSight-7B, a fine-tuned vision-language model optimized for UI element interaction, trained using LoRA on a web-focused subset of the Wave-UI-25K dataset. WebSight integrates this model into a modular multi-agent architecture, comprising planning, reasoning, vision-action, and verification agents, coordinated through an episodic memory mechanism. WebSight-7B achieves a top-1 accuracy of 58.84% on the Showdown Clicks benchmark, outperforming several larger generalist models while maintaining lower latency. The full WebSight agent achieves a 68.0% success rate on the WebVoyager benchmark, surpassing systems from labs such as OpenAI (61.0%) and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
