DesignPref: Capturing Personal Preferences in Visual Design Generation

TL;DR

This paper introduces DesignPref, a dataset of 12,000 UI design preference comparisons annotated by professional designers, revealing high disagreement levels and demonstrating personalized models outperform aggregated ones in capturing individual preferences.

Contribution

The paper presents the first dataset for personalized visual design preferences and explores strategies to model individual tastes, improving prediction accuracy over traditional methods.

Findings

High disagreement among designers (Krippendorff's alpha = 0.25)

Personalized models outperform aggregated models in preference prediction

Effective personalization achieved with significantly fewer examples

Abstract

Generative models, such as large language models and text-to-image diffusion models, are increasingly used to create visual designs like user interfaces (UIs) and presentation slides. Finetuning and benchmarking these generative models have often relied on datasets of human-annotated design preferences. Yet, due to the subjective and highly personalized nature of visual design, preference varies widely among individuals. In this paper, we study this problem by introducing DesignPref, a dataset of 12k pairwise comparisons of UI design generation annotated by 20 professional designers with multi-level preference ratings. We found that among trained designers, substantial levels of disagreement exist (Krippendorff's alpha = 0.25 for binary preferences). Natural language rationales provided by these designers indicate that disagreements stem from differing perceptions of various design aspect importance and individual preferences. With DesignPref, we demonstrate that traditional majority-voting methods for training aggregated judge models often do not accurately reflect individual preferences. To address this challenge, we investigate multiple personalization strategies, particularly fine-tuning or incorporating designer-specific annotations into RAG pipelines. Our results show that personalized models consistently outperform aggregated baseline models in predicting individual designers' preferences, even when using 20 times fewer examples. Our work provides the first dataset to study personalized visual design evaluation and support future research into modeling individual design taste.