Toward the Automated Localization of Buggy Mobile App UIs from Bug Descriptions

TL;DR

This paper explores automating the localization of buggy UI screens and components in mobile apps using deep learning techniques, demonstrating that combining visual and textual data improves bug localization accuracy and aids in fixing bugs.

Contribution

It is the first comprehensive study evaluating deep learning models for automated Buggy UI Localization at screen and component levels, highlighting the benefits of multi-modal approaches.

Findings

Models using visual info excel at UI screen localization.

Text-based models perform better for UI component localization.

Localized buggy UIs improve code bug localization by 9-12%.

Abstract

Bug report management is a costly software maintenance process comprised of several challenging tasks. Given the UI-driven nature of mobile apps, bugs typically manifest through the UI, hence the identification of buggy UI screens and UI components (Buggy UI Localization) is important to localizing the buggy behavior and eventually fixing it. However, this task is challenging as developers must reason about bug descriptions (which are often low-quality), and the visual or code-based representations of UI screens. This paper is the first to investigate the feasibility of automating the task of Buggy UI Localization through a comprehensive study that evaluates the capabilities of one textual and two multi-modal deep learning (DL) techniques and one textual unsupervised technique. We evaluate such techniques at two levels of granularity, Buggy UI Screen and UI Component localization. Our results illustrate the individual strengths of models that make use of different representations, wherein models that incorporate visual information perform better for UI screen localization, and models that operate on textual screen information perform better for UI component localization -- highlighting the need for a localization approach that blends the benefits of both types of techniques. Furthermore, we study whether Buggy UI Localization can improve traditional buggy code localization, and find that incorporating localized buggy UIs leads to improvements of 9%-12% in Hits@10.