UI Layers Group Detector: Grouping UI Layers via Text Fusion and Box Attention

TL;DR

This paper introduces a vision-based method for automatically detecting and grouping UI layers in design drafts using text fusion and box attention, improving the quality of generated UI code.

Contribution

It presents a novel UI layers grouping detector with text fusion and box attention components, and constructs a large-scale dataset with data augmentation for training.

Findings

Achieves decent accuracy in layers grouping

Utilizes text information for improved detection

Introduces a large-scale UI dataset and augmentation methods

Abstract

Graphic User Interface (GUI) is facing great demand with the popularization and prosperity of mobile apps. Automatic UI code generation from UI design draft dramatically simplifies the development process. However, the nesting layer structure in the design draft affects the quality and usability of the generated code. Few existing GUI automated techniques detect and group the nested layers to improve the accessibility of generated code. In this paper, we proposed our UI Layers Group Detector as a vision-based method that automatically detects images (i.e., basic shapes and visual elements) and text layers that present the same semantic meanings. We propose two plug-in components, text fusion and box attention, that utilize text information from design drafts as a priori information for group localization. We construct a large-scale UI dataset for training and testing, and present a data augmentation approach to boost the detection performance. The experiment shows that the proposed method achieves a decent accuracy regarding layers grouping.