IW-Bench: Evaluating Large Multimodal Models for Converting Image-to-Web

TL;DR

This paper introduces IW-Bench, a comprehensive benchmark for evaluating large multimodal models' ability to convert images into web pages, focusing on element accuracy and layout fidelity, with new prompting techniques.

Contribution

It proposes novel metrics for element and layout accuracy, curates a large benchmark dataset, and introduces a five-hop Chain-of-Thought prompting method for improved performance.

Findings

Existing models show room for improvement in element completeness.

Layout accuracy remains a challenge for current models.

The benchmark provides a new standard for evaluating image-to-web conversion.

Abstract

Recently advancements in large multimodal models have led to significant strides in image comprehension capabilities. Despite these advancements, there is a lack of the robust benchmark specifically for assessing the Image-to-Web conversion proficiency of these large models. Primarily, it is essential to ensure the integrity of the web elements generated. These elements comprise visible and invisible categories. Previous evaluation methods (e.g.,BLEU) are notably susceptible to significant alterations due to the presence of invisible elements in Web. Furthermore, it is crucial to measure the layout information of web pages, referring to the positional relationships between elements, which is overlooked by previous work. To address challenges, we have curated and aligned a benchmark of images and corresponding web codes (IW-BENCH). Specifically, we propose the Element Accuracy, which tests the completeness of the elements by parsing the Document Object Model (DOM) tree. Layout Accuracy is also proposed to analyze the positional relationships of elements by converting DOM tree into a common subsequence. Besides, we design a five-hop multimodal Chain-of-Thought Prompting for better performance, which contains five hop: 1) SoM prompt injection. 2) Inferring Elements. 3) Inferring Layout. 4) Inferring Web code. 5) Reflection. Our benchmark comprises 1200 pairs of images and web codes with varying levels of difficulty. We have conducted extensive experiments on existing large multimodal models, offering insights into their performance and areas for improvement in image-to-web domain.