Creating User Interface Mock-ups from High-Level Text Descriptions with Deep-Learning Models

TL;DR

This paper presents three deep-learning techniques to automatically generate low-fidelity UI mock-ups from high-level text descriptions, aiming to streamline the UI design process for app developers.

Contribution

It introduces two retrieval-based and one generative deep-learning methods, along with processing techniques, for translating high-level design goals into concrete UI mock-ups.

Findings

Methods produce coherent, diverse, and relevant mock-ups

Designers responded positively to the proposed techniques

Quantitative and qualitative evaluations demonstrate effectiveness

Abstract

The design process of user interfaces (UIs) often begins with articulating high-level design goals. Translating these high-level design goals into concrete design mock-ups, however, requires extensive effort and UI design expertise. To facilitate this process for app designers and developers, we introduce three deep-learning techniques to create low-fidelity UI mock-ups from a natural language phrase that describes the high-level design goal (e.g. "pop up displaying an image and other options"). In particular, we contribute two retrieval-based methods and one generative method, as well as pre-processing and post-processing techniques to ensure the quality of the created UI mock-ups. We quantitatively and qualitatively compare and contrast each method's ability in suggesting coherent, diverse and relevant UI design mock-ups. We further evaluate these methods with 15 professional UI designers and practitioners to understand each method's advantages and disadvantages. The designers responded positively to the potential of these methods for assisting the design process.