Efficient Incremental Code Coverage Analysis for Regression Test Suites

TL;DR

This paper introduces iJaCoCo, an incremental code coverage analysis technique that efficiently updates coverage data after code changes by executing only necessary tests, significantly reducing analysis time.

Contribution

The paper presents the first incremental code coverage analysis method compatible with regression test selection, implemented in the iJaCoCo tool for Java.

Findings

iJaCoCo speeds up coverage analysis by up to 8.20x

It reduces analysis time by an average of 1.86x

Evaluated on 22 open-source repositories with 1,122 versions

Abstract

Code coverage analysis has been widely adopted in the continuous integration of open-source and industry software repositories to monitor the adequacy of regression test suites. However, computing code coverage can be costly, introducing significant overhead during test execution. Plus, re-collecting code coverage for the entire test suite is usually unnecessary when only a part of the coverage data is affected by code changes. While regression test selection (RTS) techniques exist to select a subset of tests whose behaviors may be affected by code changes, they are not compatible with code coverage analysis techniques -- that is, simply executing RTS-selected tests leads to incorrect code coverage results. In this paper, we present the first incremental code coverage analysis technique, which speeds up code coverage analysis by executing a minimal subset of tests to update the coverage data affected by code changes. We implement our technique in a tool dubbed iJaCoCo, which builds on Ekstazi and JaCoCo -- the state-of-the-art RTS and code coverage analysis tools for Java. We evaluate iJaCoCo on 1,122 versions from 22 open-source repositories and show that iJaCoCo can speed up code coverage analysis time by an average of 1.86x and up to 8.20x compared to JaCoCo.