Owl Eyes: Spotting UI Display Issues via Visual Understanding
Zhe Liu, Chunyang Chen, Junjie Wang, Yuekai Huang, Jun Hu, Qing, Wang
TL;DR
OwlEyes is a deep learning-based system that detects and localizes GUI display issues in screenshots, improving bug detection and aiding developers in fixing visual bugs across devices.
Contribution
The paper introduces OwlEye, a novel deep learning approach with a large-scale dataset for detecting and localizing GUI display issues, outperforming existing methods.
Findings
Achieves 85% precision and 84% recall in issue detection.
Localizes issues with 90% accuracy.
Uncovered 57 previously undetected UI issues in real apps.
Abstract
Graphical User Interface (GUI) provides a visual bridge between a software application and end users, through which they can interact with each other. With the development of technology and aesthetics, the visual effects of the GUI are more and more attracting. However, such GUI complexity posts a great challenge to the GUI implementation. According to our pilot study of crowdtesting bug reports, display issues such as text overlap, blurred screen, missing image always occur during GUI rendering on different devices due to the software or hardware compatibility. They negatively influence the app usability, resulting in poor user experience. To detect these issues, we propose a novel approach, OwlEye, based on deep learning for modelling visual information of the GUI screenshot. Therefore, OwlEye can detect GUIs with display issues and also locate the detailed region of the issue in the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.