When Automated Program Repair Meets Regression Testing -- An Extensive Study on 2 Million Patches

TL;DR

This study systematically evaluates the effectiveness of regression test selection techniques in reducing the testing effort of automated program repair systems across 2 million patches, providing practical guidelines for integration.

Contribution

It is the first extensive analysis of RTS techniques at multiple levels for numerous APR systems, demonstrating their impact on patch validation efficiency.

Findings

RTS can significantly reduce testing time for APR patches.

Different RTS levels have varying effectiveness depending on the system.

Practical guidelines are provided for integrating RTS into APR workflows.

Abstract

In recent years, Automated Program Repair (APR) has been extensively studied in academia and even drawn wide attention from industry. However, APR techniques can be extremely time consuming since (1) a large number of patches can be generated for a given bug, and (2) each patch needs to be executed on the original tests to ensure its correctness. In the literature, various techniques (e.g., based on learning, mining, and constraint solving) have been proposed/studied to reduce the number of patches. Intuitively, every patch can be treated as a software revision during regression testing; thus, traditional Regression Test Selection (RTS) techniques can be leveraged to only execute the tests affected by each patch (as the other tests would keep the same outcomes) to further reduce patch execution time. However, few APR systems actually adopt RTS and there is still a lack of systematic studies demonstrating the benefits of RTS and the impact of different RTS strategies on APR. To this end, this paper presents the first extensive study of widely-used RTS techniques at different levels (i.e., class/method/statement levels) for 12 state-of-the-art APR systems on over 2M patches. Our study reveals various practical guidelines for bridging the gap between APR and regression testing.