Yet Another Combination of IR- and Neural-based Comment Generation

TL;DR

This paper introduces a dynamic method combining IR- and neural-based techniques for code comment generation, significantly improving performance over existing static combinations by adaptively selecting the best approach for each code snippet.

Contribution

The paper proposes a novel dynamic combination approach using a classifier to select between IR- and neural-based methods, enhancing comment generation accuracy.

Findings

Achieved a BLEU score of 25.45, outperforming previous methods.

Improved state-of-the-art results by 41% over IR-based, 26% over neural-based, and 7% over static combinations.

Demonstrated effectiveness on a large-scale Java dataset.

Abstract

Code comment generation techniques aim to generate natural language descriptions for source code. There are two orthogonal approaches for this task, i.e., information retrieval (IR) based and neural-based methods. Recent studies have focused on combining their strengths by feeding the input code and its similar code snippets retrieved by the IR-based approach to the neural-based approach, which can enhance the neural-based approach's ability to output low-frequency words and further improve the performance. However, despite the tremendous progress, our pilot study reveals that the current combination is not generalizable and can lead to performance degradation. In this paper, we propose a straightforward but effective approach to tackle the issue of existing combinations of these two comment generation approaches. Instead of binding IR- and neural-based approaches statically, we combine them in a dynamic manner. Specifically, given an input code snippet, we first use an IR-based technique to retrieve a similar code snippet from the corpus. Then we use a Cross-Encoder based classifier to decide the comment generation method to be used dynamically, i.e., if the retrieved similar code snippet is a true positive (i.e., is semantically similar to the input), we directly use the IR-based technique. Otherwise, we pass the input to the neural-based model to generate the comment. We evaluate our approach on a large-scale dataset of Java projects. Experiment results show that our approach can achieve 25.45 BLEU score, which improves the state-of-the-art IR-based approach, neural-based approach, and their combination by 41%, 26%, and 7%, respectively. We propose a straightforward but effective dynamic combination of IR-based and neural-based comment generation, which outperforms state-of-the-art approaches by a substantial margin.