Consistent or Sensitive? Automated Code Revision Tools Against Semantics-Preserving Perturbations

TL;DR

This study evaluates the consistency of automated code revision tools when faced with semantically equivalent code variants, revealing significant performance drops and highlighting the challenge of maintaining semantic stability in automated code editing.

Contribution

The paper introduces a systematic evaluation of ACR tools' consistency using semantics-preserving perturbations and demonstrates their vulnerability to such perturbations, an area previously underexplored.

Findings

ACR tools' revision accuracy drops up to 45.3% with semantically equivalent code.

Closer perturbations to targeted regions increase failure likelihood.

Attention-guiding heuristics offer only marginal improvements.

Abstract

Automated Code Revision (ACR) tools aim to reduce manual effort by automatically generating code revisions based on reviewer feedback. While ACR tools have shown promising performance on historical data, their real-world utility depends on their ability to handle similar code variants expressing the same issue - a property we define as consistency. However, the probabilistic nature of ACR tools often compromises consistency, which may lead to divergent revisions even for semantically equivalent code variants. In this paper, we investigate the extent to which ACR tools maintain consistency when presented with semantically equivalent code variants. To do so, we first designed nine types of semantics-preserving perturbations (SPP) and applied them to 2032 Java methods from real-world GitHub projects, generating over 10K perturbed variants for evaluation. Then we used these perturbations to evaluate the consistency of five state-of-the-art transformer-based ACR tools. We found that the ACR tools' ability to generate correct revisions can drop by up to 45.3%, when presented with semantically equivalent code. The closer the perturbation is to this targeted region, the more likely an ACR tool is to fail to generate the correct revision. We explored potential mitigation strategies that modify the input representation, but found that these attention-guiding heuristics yielded only marginal improvements, thus leaving the solution to this problem as an open research question.