Issue Retrieval and Verification Enhanced Supplementary Code Comment Generation

TL;DR

This paper introduces IsComment, a retrieval and verification approach that leverages issue reports to generate more accurate and comprehensive supplementary code comments, reducing hallucinations and improving reliability.

Contribution

It proposes a novel issue-based retrieval and filtering method for code comment generation that enhances comment relevance and reduces hallucinations compared to existing LLM approaches.

Findings

Coverage of manual supplementary comments increased significantly for ChatGPT, GPT-4o, and DeepSeek-V3.

IsComment generates richer, more useful comments for programming understanding.

Quantitative evaluation shows improved MESIA scores with IsComment.

Abstract

Issue reports have been recognized to contain rich information for retrieval-augmented code comment generation. However, how to minimize hallucinations in the generated comments remains significant challenges. In this paper, we propose IsComment, an issue-based LLM retrieval and verification approach for generating method's design rationale, usage directives, and so on as supplementary code comments. We first identify five main types of code supplementary information that issue reports can provide through code-comment-issue analysis. Next, we retrieve issue sentences containing these types of supplementary information and generate candidate code comments. To reduce hallucinations, we filter out those candidate comments that are irrelevant to the code or unverifiable by the issue report, making the code comment generation results more reliable. Our experiments indicate that compared with LLMs, IsComment increases the coverage of manual supplementary comments from 33.6% to 72.2% for ChatGPT, from 35.8% to 88.4% for GPT-4o, and from 35.0% to 86.2% for DeepSeek-V3. Compared with existing work, IsComment can generate richer and more useful supplementary code comments for programming understanding, which is quantitatively evaluated through the MESIA metric on both methods with and without manual code comments.