COMCAT: Leveraging Human Judgment to Improve Automatic Documentation and Summarization

TL;DR

COMCAT enhances code comprehension by automatically generating relevant comments using augmented large language models, significantly outperforming standard methods and matching human quality in a variety of software engineering tasks.

Contribution

This paper introduces COMCAT, a novel LLM-based approach that intelligently generates and selects comments to improve software understanding, with a new dataset for further research.

Findings

COMCAT-generated comments significantly improve developer comprehension by up to 12%.

Participants preferred COMCAT comments over standard ChatGPT comments in 92% of cases.

COMCAT comments are as accurate and readable as human comments.

Abstract

Software maintenance constitutes a substantial portion of the total lifetime costs of software, with a significant portion attributed to code comprehension. Software comprehension is eased by documentation such as comments that summarize and explain code. We present COMCAT, an approach to automate comment generation by augmenting Large Language Models (LLMs) with expertise-guided context to target the annotation of source code with comments that improve comprehension. Our approach enables the selection of the most relevant and informative comments for a given snippet or file containing source code. We develop the COMCAT pipeline to comment C/C++ files by (1) automatically identifying suitable locations in which to place comments, (2) predicting the most helpful type of comment for each location, and (3) generating a comment based on the selected location and comment type. In a human subject evaluation, we demonstrate that COMCAT-generated comments significantly improve developer code comprehension across three indicative software engineering tasks by up to 12% for 87% of participants. In addition, we demonstrate that COMCAT-generated comments are at least as accurate and readable as human-generated comments and are preferred over standard ChatGPT-generated comments for up to 92% of snippets of code. Furthermore, we develop and release a dataset containing source code snippets, human-written comments, and human-annotated comment categories. COMCAT leverages LLMs to offer a significant improvement in code comprehension across a variety of human software engineering tasks.