# Fine-grained Code Coverage Measurement in Automated Black-box Android Testing

## TL;DR

This paper introduces ACVTool, a tool for precise, fine-grained code coverage measurement in black-box Android testing, improving fault detection and evaluation of testing frameworks.

## Contribution

ACVTool is the first tool to reliably measure class, method, and instruction-level coverage in black-box Android app testing with minimal overhead.

## Key findings

- ACVTool successfully instruments 96.9% of tested apps.
- Fine-grained coverage uncovers more faults than coarser metrics.
- Overhead introduced by ACVTool is negligible for automated testing.

## Abstract

Today, there are millions of third-party Android applications. Some of these applications are buggy or even malicious. To identify such applications, novel frameworks for automated black-box testing and dynamic analysis are being developed by the Android community, including Google. Code coverage is one of the most common metrics for evaluating effectiveness of these frameworks. Furthermore, code coverage is used as a fitness function for guiding evolutionary and fuzzy testing techniques. However, there are no reliable tools for measuring fine-grained code coverage in black-box Android app testing. We present the Android Code coVerage Tool, ACVTool for short, that instruments Android apps and measures the code coverage in the black-box setting at the class, method and instruction granularities. ACVTool has successfully instrumented 96.9% of apps in our experiments. It introduces a negligible instrumentation time overhead, and its runtime overhead is acceptable for automated testing tools. We show in a large-scale experiment with Sapienz, a state-of-art testing tool, that the fine-grained instruction-level code coverage provided by ACVTool helps to uncover a larger amount of faults than coarser-grained code coverage metrics.