Fragmented Layer Grouping in GUI Designs Through Graph Learning Based on Multimodal Information

TL;DR

This paper introduces a graph learning approach utilizing multimodal information to automatically group fragmented GUI layers, improving code readability and maintainability in GUI design automation.

Contribution

It proposes a novel graph neural network model with self-attention for multimodal GUI layer grouping, advancing automation in GUI-to-code processes.

Findings

Achieves state-of-the-art performance on real-world datasets

Enhances GUI code readability and maintainability

Validated by user study showing practical benefits

Abstract

Automatically constructing GUI groups of different granularities constitutes a critical intelligent step towards automating GUI design and implementation tasks. Specifically, in the industrial GUI-to-code process, fragmented layers may decrease the readability and maintainability of generated code, which can be alleviated by grouping semantically consistent fragmented layers in the design prototypes. This study aims to propose a graph-learning-based approach to tackle the fragmented layer grouping problem according to multi-modal information in design prototypes. Our graph learning module consists of self-attention and graph neural network modules. By taking the multimodal fused representation of GUI layers as input, we innovatively group fragmented layers by classifying GUI layers and regressing the bounding boxes of the corresponding GUI components simultaneously. Experiments on two real-world datasets demonstrate that our model achieves state-of-the-art performance. A further user study is also conducted to validate that our approach can assist an intelligent downstream tool in generating more maintainable and readable front-end code.