Taming Android Fragmentation through Lightweight Crowdsourced Testing

TL;DR

LAZYCOW is a lightweight crowdsourced testing approach that effectively detects a broad range of Android compatibility issues on real devices, addressing the limitations of existing tools focused mainly on API signature evolution.

Contribution

The paper introduces LAZYCOW, a novel crowdsourced testing method that broadens compatibility issue detection beyond API changes, improving Android app testing on diverse devices.

Findings

LAZYCOW successfully detects various compatibility issues on real devices.

User satisfaction surveys indicate LAZYCOW is practical and well-received.

Experimental results show high effectiveness in identifying API-induced issues.

Abstract

Android fragmentation refers to the overwhelming diversity of Android devices and OS versions. These lead to the impossibility of testing an app on every supported device, leaving a number of compatibility bugs scattered in the community and thereby resulting in poor user experiences. To mitigate this, our fellow researchers have designed various works to automatically detect such compatibility issues. However, the current state-of-the-art tools can only be used to detect specific kinds of compatibility issues (i.e., compatibility issues caused by API signature evolution), i.e., many other essential types of compatibility issues are still unrevealed. For example, customized OS versions on real devices and semantic changes of OS could lead to serious compatibility issues, which are non-trivial to be detected statically. To this end, we propose a novel, lightweight, crowdsourced testing approach, LAZYCOW, to fill this research gap and enable the possibility of taming Android fragmentation through crowdsourced efforts. Specifically, crowdsourced testing is an emerging alternative to conventional mobile testing mechanisms that allow developers to test their products on real devices to pinpoint platform-specific issues. Experimental results on thousands of test cases on real-world Android devices show that LAZYCOW is effective in automatically identifying and verifying API-induced compatibility issues. Also, after investigating the user experience through qualitative metrics, users' satisfaction provides strong evidence that LAZYCOW is useful and welcome in practice.