UI Semantic Group Detection: Grouping UI Elements with Similar Semantics in Mobile Graphical User Interface
Shuhong Xiao, Yunnong Chen, Yaxuan Song, Liuqing Chen, Lingyun Sun,, Yankun Zhen, Yanfang Chang

TL;DR
This paper introduces a deep learning-based method for detecting semantic groups of UI elements in mobile app interfaces, enhancing multiple UI-related tasks like accessibility and code generation.
Contribution
It proposes UISCGD, a novel vision detector that extends deformable-DETR with UI-specific features, trained on a large dataset of mobile UI screenshots.
Findings
Achieves 6.1% better accuracy than the best baseline.
Outperforms deformable-DETR by 5.4%.
Applicable to multiple UI-related software tasks.
Abstract
Texts, widgets, and images on a UI page do not work separately. Instead, they are partitioned into groups to achieve certain interaction functions or visual information. Existing studies on UI elements grouping mainly focus on a specific single UI-related software engineering task, and their groups vary in appearance and function. In this case, we propose our semantic component groups that pack adjacent text and non-text elements with similar semantics. In contrast to those task-oriented grouping methods, our semantic component group can be adopted for multiple UI-related software tasks, such as retrieving UI perceptual groups, improving code structure for automatic UI-to-code generation, and generating accessibility data for screen readers. To recognize semantic component groups on a UI page, we propose a robust, deep learning-based vision detector, UISCGD, which extends the SOTA…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsInnovative Human-Technology Interaction · Human Mobility and Location-Based Analysis · Mobile and Web Applications
