On the Relevance of Cross-project Learning with Nearest Neighbours for Commit Message Generation

TL;DR

This paper investigates commit message generation, revealing that a simple nearest neighbor method can outperform neural machine translation techniques, especially when enhanced with cross-project learning.

Contribution

It introduces a simpler, faster variation of NNGen that outperforms the original in BLEU_4 score without relying on cross-project learning.

Findings

NNGen outperforms existing NMT-based methods

Cross-project learning does not significantly improve NNGen

A new simple variation of NNGen outperforms the original

Abstract

Commit messages play an important role in software maintenance and evolution. Nonetheless, developers often do not produce high-quality messages. A number of commit message generation methods have been proposed in recent years to address this problem. Some of these methods are based on neural machine translation (NMT) techniques. Studies show that the nearest neighbor algorithm (NNGen) outperforms existing NMT-based methods, although NNGen is simpler and faster than NMT. In this paper, we show that NNGen does not take advantage of cross-project learning in the majority of the cases. We also show that there is an even simpler and faster variation of the existing NNGen method which outperforms it in terms of the BLEU_4 score without using cross-project learning.