Improving Retrieval-Augmented Code Comment Generation by Retrieving for Generation

TL;DR

This paper introduces JOINTCOM, a retrieval-augmented code comment generation method that learns to retrieve exemplars optimized for comment quality, significantly improving performance over existing approaches.

Contribution

It proposes a novel training strategy for retrieval-augmented comment generation that aligns retriever feedback with generator performance, enhancing comment quality.

Findings

JOINTCOM outperforms state-of-the-art baselines by 7.3% to 30.0% on two datasets.

Human evaluation shows JOINTCOM produces more natural, informative, and useful comments.

The approach effectively learns to retrieve exemplars that improve comment generation.

Abstract

Code comment generation aims to generate high-quality comments from source code automatically and has been studied for years. Recent studies proposed to integrate information retrieval techniques with neural generation models to tackle this problem, i.e., Retrieval-Augmented Comment Generation (RACG) approaches, and achieved state-of-the-art results. However, the retrievers in previous work are built independently of their generators. This results in that the retrieved exemplars are not necessarily the most useful ones for generating comments, limiting the performance of existing approaches. To address this limitation, we propose a novel training strategy to enable the retriever to learn from the feedback of the generator and retrieve exemplars for generation. Specifically, during training, we use the retriever to retrieve the top-k exemplars and calculate their retrieval scores, and use the generator to calculate a generation loss for the sample based on each exemplar. By aligning high-score exemplars retrieved by the retriever with low-loss exemplars observed by the generator, the retriever can learn to retrieve exemplars that can best improve the quality of the generated comments. Based on this strategy, we propose a novel RACG approach named JOINTCOM and evaluate it on two real-world datasets, JCSD and PCSD. The experimental results demonstrate that our approach surpasses the state-of-the-art baselines by 7.3% to 30.0% in terms of five metrics on the two datasets. We also conduct a human evaluation to compare JOINTCOM with the best-performing baselines. The results indicate that JOINTCOM outperforms the baselines, producing comments that are more natural, informative, and useful.