An Empirical Study on Code Comment Completion

TL;DR

This study evaluates the effectiveness of n-gram and T5 models in assisting developers with code comment completion, revealing T5's superior performance in a large-scale empirical assessment.

Contribution

It provides an empirical comparison of simple n-gram and advanced T5 models for code comment completion, highlighting the potential of T5 in developer support tools.

Findings

T5 outperforms n-gram in comment completion accuracy

Both models significantly speed up comment writing process

T5 shows promise for integration into IDEs for real-time assistance

Abstract

Code comments play a prominent role in program comprehension activities. However, source code is not always documented and code and comments not always co-evolve. To deal with these issues, researchers have proposed techniques to automatically generate comments documenting a given code at hand. The most recent works in the area applied deep learning (DL) techniques to support such a task. Despite the achieved advances, the empirical evaluations of these approaches show that they are still far from a performance level that would make them valuable for developers. We tackle a simpler and related problem: Code comment completion. Instead of generating a comment for a given code from scratch, we investigate the extent to which state-of-the-art techniques can help developers in writing comments faster. We present a large-scale study in which we empirically assess how a simple n-gram model and the recently proposed Text-To-Text Transfer Transformer (T5) architecture can perform in autocompleting a code comment the developer is typing. The achieved results show the superiority of the T5 model, despite the n-gram model being a competitive solution.