MESIA: Understanding and Leveraging Supplementary Nature of Method-level Comments for Automatic Comment Generation

TL;DR

This paper introduces MESIA, a metric to measure the supplementary information in method comments, and demonstrates how training data quality impacts neural models' ability to generate more informative comments.

Contribution

The paper proposes the MESIA metric for assessing comment informativeness and shows how data quality influences neural comment generation capabilities.

Findings

Small-MESIA comments are about 20% of dataset and mainly 'WHAT' comments.

Large-MESIA comments are harder for neural models to generate.

Reducing small-MESIA comments in training improves large-MESIA comment generation.

Abstract

Code comments are important for developers in program comprehension. In scenarios of comprehending and reusing a method, developers expect code comments to provide supplementary information beyond the method signature. However, the extent of such supplementary information varies a lot in different code comments. In this paper, we raise the awareness of the supplementary nature of method-level comments and propose a new metric named MESIA (Mean Supplementary Information Amount) to assess the extent of supplementary information that a code comment can provide. With the MESIA metric, we conduct experiments on a popular code-comment dataset and three common types of neural approaches to generate method-level comments. Our experimental results demonstrate the value of our proposed work with a number of findings. (1) Small-MESIA comments occupy around 20% of the dataset and mostly fall into only the WHAT comment category. (2) Being able to provide various kinds of essential information, large-MESIA comments in the dataset are difficult for existing neural approaches to generate. (3) We can improve the capability of existing neural approaches to generate large-MESIA comments by reducing the proportion of small-MESIA comments in the training set. (4) The retrained model can generate large-MESIA comments that convey essential meaningful supplementary information for methods in the small-MESIA test set, but will get a lower BLEU score in evaluation. These findings indicate that with good training data, auto-generated comments can sometimes even surpass human-written reference comments, and having no appropriate ground truth for evaluation is an issue that needs to be addressed by future work on automatic comment generation.