On the need to perform comprehensive evaluations of automated program repair benchmarks: Sorald case study

TL;DR

This paper highlights the importance of comprehensive evaluation of automated program repair tools, demonstrating through a case study that current assessments overlook critical side effects like new faults and code degradation.

Contribution

It introduces a framework for evaluating APR tools holistically and applies it to Sorald, revealing significant side effects not captured by traditional metrics.

Findings

Sorald fixed some violations but introduced 2,120 new faults.

Unit test failure rate increased by 24% after repairs.

Code structure was degraded, emphasizing the need for comprehensive evaluation.

Abstract

In supporting the development of high-quality software, especially necessary in the era of LLMs, automated program repair (APR) tools aim to improve code quality by automatically addressing violations detected by static analysis profilers. Previous research tends to evaluate APR tools only for their ability to clear violations, neglecting their potential introduction of new (sometimes severe) violations, changes to code functionality and degrading of code structure. There is thus a need for research to develop and assess comprehensive evaluation frameworks for APR tools. This study addresses this research gap, and evaluates Sorald (a state-of-the-art APR tool) as a proof of concept. Sorald's effectiveness was evaluated in repairing 3,529 SonarQube violations across 30 rules within 2,393 Java code snippets extracted from Stack Overflow. Outcomes show that while Sorald fixes specific rule violations, it introduced 2,120 new faults (32 bugs, 2088 code smells), reduced code functional correctness--as evidenced by a 24% unit test failure rate--and degraded code structure, demonstrating the utility of our framework. Findings emphasize the need for evaluation methodologies that capture the full spectrum of APR tool effects, including side effects, to ensure their safe and effective adoption.