Automatically Detecting API-induced Compatibility Issues in Android Apps: A Comparative Analysis (Replicability Study)

TL;DR

This study reviews, reproduces, and empirically compares nine approaches for detecting Android app compatibility issues, revealing significant differences in their capabilities and emphasizing the need for further research in this area.

Contribution

It provides a comprehensive literature review, reproduces existing methods, and empirically compares their effectiveness on real-world datasets, highlighting gaps in current solutions.

Findings

Reproduced existing tools successfully

Limited overlap in detected issues among tools

Need for improved compatibility detection methods

Abstract

Fragmentation is a serious problem in the Android ecosystem. This problem is mainly caused by the fast evolution of the system itself and the various customizations independently maintained by different smartphone manufacturers. Many efforts have attempted to mitigate its impact via approaches to automatically pinpoint compatibility issues in Android apps. Unfortunately, at this stage, it is still unknown if this objective has been fulfilled, and the existing approaches can indeed be replicated and reliably leveraged to pinpoint compatibility issues in the wild. We, therefore, propose to fill this gap by first conducting a literature review within this topic to identify all the available approaches. Among the nine identified approaches, we then try our best to reproduce them based on their original datasets. After that, we go one step further to empirically compare those approaches against common datasets with real-world apps containing compatibility issues. Experimental results show that existing tools can indeed be reproduced, but their capabilities are quite distinct, as confirmed by the fact that there is only a small overlap of the results reported by the selected tools. This evidence suggests that more efforts should be spent by our community to achieve sound compatibility issues detection.