Detecting and Summarizing GUI Changes in Evolving Mobile Apps

TL;DR

This paper introduces GCAT, an automated system that uses computer vision and natural language processing to detect, classify, and summarize GUI changes in mobile apps, aiding developers in understanding app evolution.

Contribution

The paper presents GCAT, a novel automated approach combining computer vision and natural language generation to detect and summarize GUI changes in mobile app evolution.

Findings

GCAT outperforms developers in detecting GUI changes.

GCAT achieves high precision and recall in change detection.

Generated summaries are useful for program comprehension.

Abstract

Mobile applications have become a popular software development domain in recent years due in part to a large user base, capable hardware, and accessible platforms. However, mobile developers also face unique challenges, including pressure for frequent releases to keep pace with rapid platform evolution, hardware iteration, and user feedback. Due to this rapid pace of evolution, developers need automated support for documenting the changes made to their apps in order to aid in program comprehension. One of the more challenging types of changes to document in mobile apps are those made to the graphical user interface (GUI) due to its abstract, pixel-based representation. In this paper, we present a fully automated approach, called GCAT, for detecting and summarizing GUI changes during the evolution of mobile apps. GCAT leverages computer vision techniques and natural language generation to accurately and concisely summarize changes made to the GUI of a mobile app between successive commits or releases. We evaluate the performance of our approach in terms of its precision and recall in detecting GUI changes compared to developer specified changes, and investigate the utility of the generated change reports in a controlled user study. Our results indicate that GCAT is capable of accurately detecting and classifying GUI changes - outperforming developers - while providing useful documentation.