Bridging Design and Development with Automated Declarative UI Code Generation

TL;DR

DeclarUI is an automated system that combines computer vision, multimodal large language models, and iterative optimization to convert UI designs into high-quality, functional declarative code across multiple mobile frameworks, significantly improving accuracy and efficiency.

Contribution

This paper introduces DeclarUI, a novel approach that integrates CV, MLLMs, and optimization to improve UI code generation from designs, surpassing existing methods in accuracy and generalizability.

Findings

Achieves 96.8% PTG coverage rate in React Native

Improves visual similarity scores by up to 55%

Increases compilation success rate by 29%

Abstract

Declarative UI frameworks have gained widespread adoption in mobile app development, offering benefits such as improved code readability and easier maintenance. Despite these advantages, the process of translating UI designs into functional code remains challenging and time-consuming. Recent advancements in multimodal large language models (MLLMs) have shown promise in directly generating mobile app code from user interface (UI) designs. However, the direct application of MLLMs to this task is limited by challenges in accurately recognizing UI components and comprehensively capturing interaction logic. To address these challenges, we propose DeclarUI, an automated approach that synergizes computer vision (CV), MLLMs, and iterative compiler-driven optimization to generate and refine declarative UI code from designs. DeclarUI enhances visual fidelity, functional completeness, and code quality through precise component segmentation, Page Transition Graphs (PTGs) for modeling complex inter-page relationships, and iterative optimization. In our evaluation, DeclarUI outperforms baselines on React Native, a widely adopted declarative UI framework, achieving a 96.8% PTG coverage rate and a 98% compilation success rate. Notably, DeclarUI demonstrates significant improvements over state-of-the-art MLLMs, with a 123% increase in PTG coverage rate, up to 55% enhancement in visual similarity scores, and a 29% boost in compilation success rate. We further demonstrate DeclarUI's generalizability through successful applications to Flutter and ArkUI frameworks.