An Analysis of the Search Spaces for Generate and Validate Patch Generation Systems

TL;DR

This paper systematically analyzes the search spaces of automatic patch generation systems, revealing the scarcity of correct patches, the abundance of passing incorrect patches, and the impact of search space size on system effectiveness.

Contribution

It provides the first comprehensive analysis of patch search space characteristics, highlighting key tradeoffs and challenges in automatic patch generation.

Findings

Correct patches are rare in search spaces.

Incorrect patches passing tests are common.

Larger search spaces can reduce the number of correct patches found.

Abstract

We present the first systematic analysis of the characteristics of patch search spaces for automatic patch generation systems. We analyze the search spaces of two current state-of-the-art systems, SPR and Prophet, with 16 different search space configurations. Our results are derived from an analysis of 1104 different search spaces and 768 patch generation executions. Together these experiments consumed over 9000 hours of CPU time on Amazon EC2. The analysis shows that 1) correct patches are sparse in the search spaces (typically at most one correct patch per search space per defect), 2) incorrect patches that nevertheless pass all of the test cases in the validation test suite are typically orders of magnitude more abundant, and 3) leveraging information other than the test suite is therefore critical for enabling the system to successfully isolate correct patches. We also characterize a key tradeoff in the structure of the search spaces. Larger and richer search spaces that contain correct patches for more defects can actually cause systems to find fewer, not more, correct patches. We identify two reasons for this phenomenon: 1) increased validation times because of the presence of more candidate patches and 2) more incorrect patches that pass the test suite and block the discovery of correct patches. These fundamental properties, which are all characterized for the first time in this paper, help explain why past systems often fail to generate correct patches and help identify challenges, opportunities, and productive future directions for the field.