A Critical Review of "Automatic Patch Generation Learned from Human-Written Patches": Essay on the Problem Statement and the Evaluation of Automatic Software Repair

TL;DR

This paper critically reviews the evaluation criteria and fundamental ideas of automatic software repair, emphasizing the importance of clear definitions and the potential contradictions in repair scenario assessments.

Contribution

It clarifies key concepts in automatic software repair and discusses the implications of evaluation criteria like correctness, understandability, and completeness.

Findings

Evaluation goals can be contradictory depending on repair scenarios

Fix acceptability is closely related to software correctness

Clarifies essential ideas behind automatic software repair

Abstract

At ICSE'2013, there was the first session ever dedicated to automatic program repair. In this session, Kim et al. presented PAR, a novel template-based approach for fixing Java bugs. We strongly disagree with key points of this paper. Our critical review has two goals. First, we aim at explaining why we disagree with Kim and colleagues and why the reasons behind this disagreement are important for research on automatic software repair in general. Second, we aim at contributing to the field with a clarification of the essential ideas behind automatic software repair. In particular we discuss the main evaluation criteria of automatic software repair: understandability, correctness and completeness. We show that depending on how one sets up the repair scenario, the evaluation goals may be contradictory. Eventually, we discuss the nature of fix acceptability and its relation to the notion of software correctness.