Scaling Up: Revisiting Mining Android Sandboxes at Scale for Malware Classification

TL;DR

This study reevaluates the effectiveness of the Mining Android Sandbox (MAS) approach for malware detection using a significantly larger dataset, revealing its limitations and the need for complementary techniques.

Contribution

It provides a large-scale replication study of the MAS approach, demonstrating its reduced performance on diverse, extensive datasets compared to prior small-scale evaluations.

Findings

MAS approach's F1-score drops from 0.90 to 0.54 with larger dataset

Certain malware families are poorly detected by MAS

Scaling reduces the effectiveness of the original method

Abstract

The widespread use of smartphones in daily life has raised concerns about privacy and security among researchers and practitioners. Privacy issues are generally highly prevalent in mobile applications, particularly targeting the Android platform, the most popular mobile operating system. For this reason, several techniques have been proposed to identify malicious behavior in Android applications, including the Mining Android Sandbox approach (MAS approach), which aims to identify malicious behavior in repackaged Android applications (apps). However, previous empirical studies evaluated the MAS approach using a small dataset consisting of only 102 pairs of original and repackaged apps. This limitation raises questions about the external validity of their findings and whether the MAS approach can be generalized to larger datasets. To address these concerns, this paper presents the results of a replication study focused on evaluating the performance of the MAS approach regarding its capabilities of correctly classifying malware from different families. Unlike previous studies, our research employs a dataset that is an order of magnitude larger, comprising 4,076 pairs of apps covering a more diverse range of Android malware families. Surprisingly, our findings indicate a poor performance of the MAS approach for identifying malware, with the F1-score decreasing from 0.90 for the small dataset used in the previous studies to 0.54 in our more extensive dataset. Upon closer examination, we discovered that certain malware families partially account for the low accuracy of the MAS approach, which fails to classify a repackaged version of an app as malware correctly. Our findings highlight the limitations of the MAS approach, particularly when scaled, and underscore the importance of complementing it with other techniques to detect a broader range of malware effectively.