Nighthawk: Fully Automated Localizing UI Display Issues via Visual Understanding

TL;DR

Nighthawk is an automated deep learning-based tool that detects and localizes UI display issues in mobile app GUIs, improving bug detection efficiency and aiding developers in fixing visual bugs.

Contribution

The paper introduces Nighthawk, a novel fully automated approach combining deep learning and heuristic data generation to detect and localize GUI display issues.

Findings

Achieves 0.84 precision and recall in issue detection.

Localizes issues with 0.59 AP and 0.60 AR.

Uncovered 151 new UI display issues in real apps.

Abstract

Graphical User Interface (GUI) provides a visual bridge between a software application and end users, through which they can interact with each other. With the upgrading of mobile devices and the development of aesthetics, the visual effects of the GUI are more and more attracting, and users pay more attention to the accessibility and usability of applications. However, such GUI complexity posts a great challenge to the GUI implementation. According to our pilot study of crowdtesting bug reports, display issues such as text overlap, component occlusion, missing image always occur during GUI rendering on different devices due to the software or hardware compatibility. They negatively influence the app usability, resulting in poor user experience. To detect these issues, we propose a fully automated approach, Nighthawk, based on deep learning for modelling visual information of the GUI screenshot. Nighthawk can detect GUIs with display issues and also locate the detailed region of the issue in the given GUI for guiding developers to fix the bug. At the same time, training the model needs a large amount of labeled buggy screenshots, which requires considerable manual effort to prepare them. We therefore propose a heuristic-based training data auto-generation method to automatically generate the labeled training data. The evaluation demonstrates that our Nighthawk can achieve average 0.84 precision and 0.84 recall in detecting UI display issues, average 0.59 AP and 0.60 AR in localizing these issues. We also evaluate Nighthawk with popular Android apps on Google Play and F-Droid, and successfully uncover 151 previously-undetected UI display issues with 75 of them being confirmed or fixed so far.