Estimating the Potential of Program Repair Search Spaces with Commit Analysis

TL;DR

This paper introduces a static, lightweight method called LighteR to evaluate the potential of program repair search spaces by analyzing past commits, enabling easier assessment without heavy configuration.

Contribution

The paper presents a novel static approach to evaluate repair search spaces by encoding repair strategies and checking commit inclusion, reducing reliance on complex tool execution.

Findings

LighteR achieves 77% precision in identifying relevant commits.

LighteR achieves 92% recall in capturing commits within repair spaces.

The method is effective across 72 GitHub repositories.

Abstract

The most natural method for evaluating program repair systems is to run them on bug datasets, such as Defects4J. Yet, using this evaluation technique on arbitrary real-world programs requires heavy configuration. In this paper, we propose a purely static method to evaluate the potential of the search space of repair approaches. This new method enables researchers and practitioners to encode the search spaces of repair approaches and select potentially useful ones without struggling with tool configuration and execution. We encode the search spaces by specifying the repair strategies they employ. Next, we use the specifications to check whether past commits lie in repair search spaces. For a repair approach, including many human-written past commits in its search space indicates its potential to generate useful patches. We implement our evaluation method in LighteR. LighteR gets a Git repository and outputs a list of commits whose source code changes lie in repair search spaces. We run LighteR on 55,309 commits from the history of 72 Github repositories with and show that LighteR's precision and recall are 77% and 92%, respectively. Overall, our experiments show that our novel method is both lightweight and effective to study the search space of program repair approaches.