Beyond Pixels: Exploring DOM Downsampling for LLM-Based Web Agents

TL;DR

This paper introduces D2Snap, a DOM downsampling algorithm that enables LLM-based web agents to efficiently process structured web page data, matching and surpassing GUI snapshot performance in task success rates.

Contribution

We propose D2Snap, the first DOM downsampling method tailored for LLM web agents, improving input efficiency while maintaining high task success rates.

Findings

D2Snap achieves a 67% success rate, comparable to GUI snapshots at 65%.

D2Snap outperforms the baseline by 8% when using larger token contexts.

DOM hierarchy is a strong UI feature for LLM understanding.

Abstract

Frontier LLMs only recently enabled serviceable, autonomous web agents. At that, a model poses as an instantaneous domain model backend. Ought to suggest interaction, it is consulted with a web-based task and respective application state. The key problem lies in application state serialisation - referred to as snapshot. State-of-the-art web agents are premised on grounded GUI snapshots, i.e., screenshots enhanced with visual cues. Not least to resemble human perception, but for images representing relatively cheap means of model input. LLM vision still lag behind code interpretation capabilities. DOM snapshots, which structurally resemble HTML, impose a desired alternative. Vast model input token size, however, disables reliable implementation with web agents to date. We propose D2Snap, a first-of-its-kind DOM downsampling algorithm. Based on a GPT-4o backend, we evaluate D2Snap on tasks sampled from the Online-Mind2Web dataset. The success rate of D2Snap-downsampled DOM snapshots (67%) matches a grounded GUI snapshot baseline (65%) - within the same input token order of magnitude (1e3). Our best evaluated configurations - one token order above, but within the model's context window - outperform this baseline by 8%. Our evaluation, moreover, yields that DOM-inherent hierarchy embodies a strong UI feature for LLMs.