XBIDetective: Leveraging Vision Language Models for Identifying Cross-Browser Visual Inconsistencies

TL;DR

XBIDetective uses vision language models to automatically identify cross-browser visual inconsistencies, improving detection accuracy for rendering bugs, dynamic elements, and advertisements across websites.

Contribution

The paper introduces XBIDetective, a novel tool leveraging vision language models for detecting cross-browser visual inconsistencies, addressing limitations of existing analysis techniques.

Findings

79% accuracy in identifying cross-browser discrepancies

84% accuracy in detecting dynamic elements

85% accuracy in detecting advertisements

Abstract

Browser rendering bugs can be challenging to detect for browser developers, as they may be triggered by very specific conditions that are exhibited on only a very small subset of websites. Cross-browser inconsistencies (XBIs), variations in how a website is interpreted and displayed on different browsers, can be helpful guides to detect such rendering bugs. Although visual and Document Object Model (DOM)-based analysis techniques exist for detecting XBIs, they often struggle with dynamic and interactive elements. In this study, we discuss our industry experience with using vision language models (VLMs) to identify XBIs. We present the XBIDetective tool which automatically captures screenshots of a website in Mozilla Firefox and Google Chrome, and analyzes them with a VLM for XBIs. We evaluate XBIDetective's performance with an off-the-shelf and a fine-tuned VLM on 1,052 websites. We show that XBIDetective can identify cross-browser discrepancies with 79% accuracy and detect dynamic elements and advertisements with 84% and 85% accuracy, respectively, when using the fine-tuned VLM. We discuss important lessons learned, and we present several potential practical use cases for XBIDetective, including automated regression testing, large-scale monitoring of websites, and rapid triaging of XBI bug reports.