Bridging Gulfs in UI Generation through Semantic Guidance

TL;DR

This paper introduces a system that uses hierarchical semantic guidance to improve user control and interpretability in AI-generated UI design, bridging the communication gap between user intent and AI output.

Contribution

It presents a novel approach that leverages explicit semantic representations to enhance control, interpretability, and refinement in AI-driven UI generation.

Findings

Users felt more in control of design intent

Improved predictability in UI refinement

Enhanced understanding of generated UIs

Abstract

While generative AI enables high-fidelity UI generation from text prompts, users struggle to articulate design intent and evaluate or refine results-creating gulfs of execution and evaluation. To understand the information needed for UI generation, we conducted a thematic analysis of UI prompting guidelines, identifying key design semantics and discovering that they are hierarchical and interdependent. Leveraging these findings, we developed a system that enables users to specify semantics, visualize relationships, and extract how semantics are reflected in generated UIs. By making semantics serve as an intermediate representation between human intent and AI output, our system bridges both gulfs by making requirements explicit and outcomes interpretable. A comparative user study suggests that our approach enhances users' perceived control over intent expression and outcome interpretation, and facilitates more predictable iterative refinement. Our work demonstrates how explicit semantic representation enables systematic and explainable exploration of design possibilities in AI-driven UI design.