Automated Reporting of GUI Design Violations for Mobile Apps

TL;DR

This paper presents GVT, an automated tool that uses computer vision to verify if mobile app GUIs match their mock-ups, improving accuracy and efficiency in design implementation verification.

Contribution

The paper introduces GVT, a novel automated approach leveraging computer vision to detect GUI design violations in mobile apps, validated through empirical and industrial evaluations.

Findings

GVT achieves high accuracy in detecting GUI violations.

The tool is scalable and useful for industrial design teams.

Over 1,000 designers and developers at Huawei use GVT.

Abstract

The inception of a mobile app often takes form of a mock-up of the Graphical User Interface (GUI), represented as a static image delineating the proper layout and style of GUI widgets that satisfy requirements. Following this initial mock-up, the design artifacts are then handed off to developers whose goal is to accurately implement these GUIs and the desired functionality in code. Given the sizable abstraction gap between mock-ups and code, developers often introduce mistakes related to the GUI that can negatively impact an app's success in highly competitive marketplaces. Moreover, such mistakes are common in the evolutionary context of rapidly changing apps. This leads to the time-consuming and laborious task of design teams verifying that each screen of an app was implemented according to intended design specifications. This paper introduces a novel, automated approach for verifying whether the GUI of a mobile app was implemented according to its intended design. Our approach resolves GUI-related information from both implemented apps and mock-ups and uses computer vision techniques to identify common errors in the implementations of mobile GUIs. We implemented this approach for Android in a tool called GVT and carried out both a controlled empirical evaluation with open-source apps as well as an industrial evaluation with designers and developers from Huawei. The results show that GVT solves an important, difficult, and highly practical problem with remarkable efficiency and accuracy and is both useful and scalable from the point of view of industrial designers and developers. The tool is currently used by over one-thousand industrial designers and developers at Huawei to improve the quality of their mobile apps.