UI Layers Merger: Merging UI layers via Visual Learning and Boundary Prior
Yun-nong Chen, Yan-kun Zhen, Chu-ning Shi, Jia-zhi Li, Liu-qing Chen,, Ze-jian Li, Ling-yun Sun, Ting-ting Zhou, Yan-fang Chang

TL;DR
This paper introduces UILM, a vision-based method that automatically detects and merges fragmented UI layers into cohesive components, improving code generation quality from UI design drafts.
Contribution
The paper presents UILM, a novel approach combining boundary-aware detection and layer merging algorithms, along with a large-scale dataset for training and evaluation.
Findings
Outperforms baseline in merging area detection accuracy
Achieves high accuracy in layers merging task
Effective dynamic data augmentation enhances detection performance
Abstract
With the fast-growing GUI development workload in the Internet industry, some work on intelligent methods attempted to generate maintainable front-end code from UI screenshots. It can be more suitable for utilizing UI design drafts that contain UI metadata. However, fragmented layers inevitably appear in the UI design drafts which greatly reduces the quality of code generation. None of the existing GUI automated techniques detects and merges the fragmented layers to improve the accessibility of generated code. In this paper, we propose UI Layers Merger (UILM), a vision-based method, which can automatically detect and merge fragmented layers into UI components. Our UILM contains Merging Area Detector (MAD) and a layers merging algorithm. MAD incorporates the boundary prior knowledge to accurately detect the boundaries of UI components. Then, the layers merging algorithm can search out…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsWeb Data Mining and Analysis · Software Engineering Research · Advanced Malware Detection Techniques
