Visual Prompting with Iterative Refinement for Design Critique Generation

TL;DR

This paper introduces an iterative visual prompting method using large language models to generate detailed, visually grounded UI design critiques, improving quality and consistency over baseline approaches.

Contribution

It presents a novel iterative visual prompting approach that leverages LLMs for high-quality, region-specific UI critique generation, demonstrating improved performance and generalizability.

Findings

Human experts preferred critiques from our pipeline over baselines.

Our method reduced the gap from human performance by 50%.

The approach outperformed baselines in object and attribute detection tasks.

Abstract

Feedback is crucial for every design process, such as user interface (UI) design, and automating design critiques can significantly improve the efficiency of the design workflow. Although existing multimodal large language models (LLMs) excel in many tasks, they often struggle with generating high-quality design critiques -- a complex task that requires producing detailed design comments that are visually grounded in a given design's image. Building on recent advancements in iterative refinement of text output and visual prompting methods, we propose an iterative visual prompting approach for UI critique that takes an input UI screenshot and design guidelines and generates a list of design comments, along with corresponding bounding boxes that map each comment to a specific region in the screenshot. The entire process is driven completely by LLMs, which iteratively refine both the text output and bounding boxes using few-shot samples tailored for each step. We evaluated our approach using Gemini-1.5-pro and GPT-4o, and found that human experts generally preferred the design critiques generated by our pipeline over those by the baseline, with the pipeline reducing the gap from human performance by 50% for one rating metric. To assess the generalizability of our approach to other multimodal tasks, we applied our pipeline to open-vocabulary object and attribute detection, and experiments showed that our method also outperformed the baseline.