Flowy: Supporting UX Design Decisions Through AI-Driven Pattern Annotation in Multi-Screen User Flows

TL;DR

Flowy is an AI-powered tool that aids UX designers in understanding and applying design patterns across multi-screen user flows, addressing limitations of existing tools that focus only on static screens.

Contribution

The paper introduces Flowy, a novel app that leverages multimodal AI models and a user flow dataset to support multi-screen UX design decision-making.

Findings

Flowy helps designers identify relevant design patterns.

User study shows Flowy supports realistic UX design tasks.

Design considerations are applicable to other creative domains.

Abstract

Many recent AI-powered UX design tools focus on generating individual static UI screens from natural language. However, they overlook the crucial aspect of interactions and user experiences across multiple screens. Through formative studies with UX professionals, we identified limitations of these tools in supporting realistic UX design workflows. In response, we designed and developed Flowy, an app that augments designers' information foraging process in ideation by supplementing specific user flow examples with distilled design pattern knowledge. Flowy utilizes large multimodal AI models and a high-quality user flow dataset to help designers identify and understand relevant abstract design patterns in the design space for multi-screen user flows. Our user study with professional UX designers demonstrates how Flowy supports realistic UX tasks. Our design considerations in Flowy, such as representations with appropriate levels of abstraction and assisted navigation through the solution space, are generalizable to other creative tasks and embody a human-centered, intelligence augmentation approach to using AI in UX design.