Automated Classification of Overfitting Patches with Statically Extracted Code Features

TL;DR

This paper introduces ODS, a static analysis and machine learning-based system for detecting overfitting patches in automatic program repair, significantly improving correctness classification accuracy.

Contribution

The paper presents a novel static feature extraction and supervised learning approach for overfitting patch detection, outperforming existing methods.

Findings

ODS correctly classifies 71.9% of patches, surpassing previous methods.

Large-scale evaluation on 10,302 patches from multiple benchmarks.

Applicable as a post-processing step for various APR systems.

Abstract

Automatic program repair (APR) aims to reduce the cost of manually fixing software defects. However, APR suffers from generating a multitude of overfitting patches, those patches that fail to correctly repair the defect beyond making the tests pass. This paper presents a novel overfitting patch detection system called ODS to assess the correctness of APR patches. ODS first statically compares a patched program and a buggy program in order to extract code features at the abstract syntax tree (AST) level. Then, ODS uses supervised learning with the captured code features and patch correctness labels to automatically learn a probabilistic model. The learned ODS model can then finally be applied to classify new and unseen program repair patches. We conduct a large-scale experiment to evaluate the effectiveness of ODS on patch correctness classification based on 10,302 patches from Defects4J, Bugs.jar and Bears benchmarks. The empirical evaluation shows that ODS is able to correctly classify 71.9% of program repair patches from 26 projects, which improves the state-of-the-art. ODS is applicable in practice and can be employed as a post-processing procedure to classify the patches generated by different APR systems.