On the Impact of Code Comments for Automated Bug-Fixing: An Empirical Study

TL;DR

This study empirically examines how the presence of comments in code affects the performance of Large Language Models in automated bug fixing, revealing that comments significantly enhance bug-fixing accuracy especially when present during both training and inference.

Contribution

It demonstrates that comments improve bug-fixing accuracy in LLMs and highlights the importance of comments detailing implementation for effective automated bug fixing.

Findings

Comments can improve bug-fixing accuracy by up to three times.

Training with comments does not harm performance when comments are absent.

Implementation-focused comments are particularly beneficial for bug fixing.

Abstract

Large Language Models (LLMs) are increasingly relevant in Software Engineering research and practice, with Automated Bug Fixing (ABF) being one of their key applications. ABF involves transforming a buggy method into its fixed equivalent. A common preprocessing step in ABF involves removing comments from code prior to training. However, we hypothesize that comments may play a critical role in fixing certain types of bugs by providing valuable design and implementation insights. In this study, we investigate how the presence or absence of comments, both during training and at inference time, impacts the bug-fixing capabilities of LLMs. We conduct an empirical evaluation comparing two model families, each evaluated under all combinations of training and inference conditions (with and without comments), and thereby revisiting the common practice of removing comments during training. To address the limited availability of comments in state-of-the-art datasets, we use an LLM to automatically generate comments for methods lacking them. Our findings show that comments improve ABF accuracy by up to threefold when present in both phases, while training with comments does not degrade performance when instances lack them. Additionally, an interpretability analysis identifies that comments detailing method implementation are particularly effective in aiding LLMs to fix bugs accurately.