Towards Reliable Evaluation of Neural Program Repair with Natural Robustness Testing

TL;DR

This paper advocates for evaluating neural program repair robustness using naturally-occurring data transformations, revealing their impact on performance and proposing an LLM-based naturalness assessment metric.

Contribution

It introduces a naturalness-focused robustness testing framework for NPR, including a human study on transformation naturalness and an LLM-based automatic assessment method.

Findings

Only 60% of transformations are natural according to human judgment.

NPR performance significantly drops on transformed datasets.

Different NPR techniques show varied robustness, indicating evaluation biases.

Abstract

In this paper, we propose shifting the focus of robustness evaluation for Neural Program Repair (NPR) techniques toward naturally-occurring data transformations. To accomplish this, we first examine the naturalness of semantic-preserving transformations through a two-stage human study. This study includes (1) interviews with senior software developers to establish concrete criteria for evaluating the naturalness of these transformations, and (2) a survey involving 10 developers to assess the naturalness of 1,178 transformations, i.e., pairs of original and transformed programs, applied to 225 real-world bugs. Our findings show that only 60% of these transformations are deemed natural, while 20% are considered unnatural, with strong agreement among annotators. Moreover, the unnaturalness of these transformations significantly impacts both their applicability to benchmarks and the conclusions drawn from robustness testing. Next, we conduct natural robustness testing on NPR techniques to assess their true effectiveness against real-world data variations. Our experimental results reveal a substantial number of prediction changes in NPR techniques, leading to significant reductions in both plausible and correct patch rates when comparing performance on the original and transformed datasets. Additionally, we observe notable differences in performance improvements between NPR techniques, suggesting potential biases on NPR evaluation introduced by limited datasets. Finally, we propose an LLM-based metric to automate the assessment of transformation naturalness, ensuring the scalability of natural robustness testing.