On the Quality of AI-Generated Source Code Comments: A Comprehensive Evaluation

TL;DR

This study evaluates the quality of AI-generated source code comments using large-scale empirical analysis, revealing that current metrics are insufficient and that richer code context improves comment quality.

Contribution

It provides a comprehensive evaluation of LLM-generated code comments, highlighting the limitations of traditional metrics and the positive impact of code complexity on comment quality.

Findings

58.8% comments were equivalent to original comments

IR metrics do not reliably reflect human judgment

Richer code context slightly improves comment quality

Abstract

This paper investigates the quality of source code comments automatically generated by Large Language Models (LLMs). While AI-based comment generation has emerged as a promising solution to reduce developers' documentation effort, prior studies have been limited by small datasets or by relying solely on traditional Information Retrieval (IR) metrics, which are insufficient to capture documentation quality. To address these limitations, we conducted a large-scale empirical study on 142 classes and 273 methods created after the training cut-off of the evaluated models. For each code element, we generated Javadoc comments using three LLMs (GPT-3.5 Turbo, GPT-4o, and DeepSeek-V3). A qualitative assessment of the comments-performed independently by two experts-showed that 58.8% were equivalent to, and 27.7% superior to, the original comments. A quantitative analysis using BLEU, ROUGE-L, and METEOR confirmed that IR-based metrics do not reliably reflect human evaluations, revealing the need for new documentation-specific metrics. Finally, correlation analyses indicated slightly positive relationships between code properties (size, complexity, coupling) and comment quality, confirming that LLMs benefit from richer contextual information.