Towards Benchmarking GUI Compatibility Testing on Mobile Applications

TL;DR

This paper introduces a benchmark for GUI compatibility testing on mobile apps, evaluating existing tools' effectiveness in test case replay and identifying causes of failure to guide future improvements.

Contribution

It presents the first benchmark for GUI compatibility testing, providing a systematic evaluation of existing tools' replay success rates and insights into their limitations.

Findings

Existing tools have varying success rates in test case replay.

Causes of replay failure include device and platform differences.

Opportunities for improving GUI compatibility testing are identified.

Abstract

GUI is a bridge connecting user and application. Existing GUI testing tasks can be categorized into two groups: functionality testing and compatibility testing. While the functionality testing focuses on detecting application runtime bugs, the compatibility testing aims at detecting bugs resulting from device or platform difference. To automate testing procedures and improve testing efficiency, previous works have proposed dozens of tools. To evaluate these tools, in functionality testing, researchers have published testing benchmarks. Comparatively, in compatibility testing, the question of ``Do existing methods indeed effectively assist test cases replay?'' is not well answered. To answer this question and advance the related research in GUI compatibility testing, we propose a benchmark of GUI compatibility testing. In our experiments, we compare the replay success rate of existing tools. Based on the experimental results, we summarize causes which may lead to ineffectiveness in test case replay and propose opportunities for improving the state-of-the-art.