On AI-Inspired UI-Design

TL;DR

This paper explores three AI-driven methods—LLMs, VLMs, and DMs—to enhance UI design creativity, offering new tools for designers to generate, search, and inspire UI concepts effectively.

Contribution

It introduces a comprehensive AI-inspired design process utilizing LLMs, VLMs, and DMs to support and inspire UI designers in creating diverse and innovative interfaces.

Findings

LLMs can generate and refine UI designs through prompts.

VLMs enable effective search within large screenshot datasets.

DLMs can produce inspirational UI images tailored for design inspiration.

Abstract

Graphical User Interface (or simply UI) is a primary mean of interaction between users and their devices. In this paper, we discuss three complementary Artificial Intelligence (AI) approaches for triggering the creativity of app designers and inspiring them create better and more diverse UI designs. First, designers can prompt a Large Language Model (LLM) to directly generate and adjust UIs. Second, a Vision-Language Model (VLM) enables designers to effectively search a large screenshot dataset, e.g. from apps published in app stores. Third, a Diffusion Model (DM) can be trained to specifically generate UIs as inspirational images. We present an AI-inspired design process and discuss the implications and limitations of the approaches.