Revision Matters: Generative Design Guided by Revision Edits

TL;DR

This paper demonstrates that incorporating human revision edits into multimodal generative models significantly improves layout design quality, approaching human performance, and highlights the importance of human guidance in iterative design refinement.

Contribution

The study introduces a dataset of human layout revisions and shows how fine-tuning a large multimodal model with this data enhances iterative design capabilities.

Findings

Human revisions improve layout quality close to human performance.

Self-revisions can cause degradation and lack of improvement.

Early-stage human guidance is crucial for effective iterative design.

Abstract

Layout design, such as user interface or graphical layout in general, is fundamentally an iterative revision process. Through revising a design repeatedly, the designer converges on an ideal layout. In this paper, we investigate how revision edits from human designer can benefit a multimodal generative model. To do so, we curate an expert dataset that traces how human designers iteratively edit and improve a layout generation with a prompted language goal. Based on such data, we explore various supervised fine-tuning task setups on top of a Gemini multimodal backbone, a large multimodal model. Our results show that human revision plays a critical role in iterative layout refinement. While being noisy, expert revision edits lead our model to a surprisingly strong design FID score ~10 which is close to human performance (~6). In contrast, self-revisions that fully rely on model's own judgement, lead to an echo chamber that prevents iterative improvement, and sometimes leads to generative degradation. Fortunately, we found that providing human guidance plays at early stage plays a critical role in final generation. In such human-in-the-loop scenario, our work paves the way for iterative design revision based on pre-trained large multimodal models.