When simplicity meets effectiveness: Detecting code comments coherence with word embeddings and LSTM

TL;DR

This paper introduces Co3D, a practical method for detecting code comment coherence using word embeddings and LSTM, emphasizing internal word meaning and order, and outperforming existing approaches.

Contribution

The work presents a simple yet effective approach combining word2vec and LSTM for code comment coherence detection, highlighting the importance of internal features over complex models.

Findings

Co3D outperforms well-established baselines in coherence detection.

Using simple architectures can achieve satisfying prediction performance.

Emphasizing internal meaning and word order improves coherence detection accuracy.

Abstract

Code comments play a crucial role in software development, as they provide programmers with practical information, allowing them to understand better the intent and semantics of the underpinning code. Nevertheless, developers tend to leave comments unchanged after updating the code, resulting in a discrepancy between the two artifacts. Such a discrepancy may trigger misunderstanding and confusion among developers, impeding various activities, including code comprehension and maintenance. Thus, it is crucial to identify if, given a code snippet, its corresponding comment is coherent and reflects well the intent behind the code. Unfortunately, existing approaches to this problem, while obtaining an encouraging performance, either rely on heavily pre-trained models, or treat input data as text, neglecting the intrinsic features contained in comments and code, including word order and synonyms. This work presents Co3D as a practical approach to the detection of code comment coherence. We pay attention to internal meaning of words and sequential order of words in text while predicting coherence in code-comment pairs. We deployed a combination of Gensim word2vec encoding and a simple recurrent neural network, a combination of Gensim word2vec encoding and an LSTM model, and CodeBERT. The experimental results show that Co3D obtains a promising prediction performance, thus outperforming well-established baselines. We conclude that depending on the context, using a simple architecture can introduce a satisfying prediction.