RAG-Enhanced Commit Message Generation

TL;DR

This paper introduces REACT, a retrieval-augmented framework that combines advanced retrieval techniques with language models to significantly improve automatic commit message generation.

Contribution

It proposes a novel hybrid retrieval method integrated with PLMs and LLMs, enhancing their performance on commit message generation tasks.

Findings

REACT improves BLEU scores of models by up to 102%.

The framework surpasses all baseline methods in experiments.

Retrieval augmentation effectively enhances code-related language tasks.

Abstract

Commit message is one of the most important textual information in software development and maintenance. However, it is time-consuming to write commit messages manually. Commit Message Generation (CMG) has become a research hotspot. Recently, several pre-trained language models (PLMs) and large language models (LLMs) with code capabilities have been introduced, demonstrating impressive performance on code-related tasks. Meanwhile, prior studies have explored the utilization of retrieval techniques for CMG, but it is still unclear what effects would emerge from combining advanced retrieval techniques with various generation models. This paper proposed REACT, a REtrieval-Augmented framework for CommiT message generation. It integrates advanced retrieval techniques with different PLMs and LLMs, to enhance the performance of these models on the CMG task. Specifically, a hybrid retriever is designed and used to retrieve the most relevant code diff and commit message pair as an exemplar. Then, the retrieved pair is utilized to guide and enhance the CMG task by PLMs and LLMs through fine-tuning and in-context learning. The experimental results show that REACT significantly enhances these models' performance on the CMG task, improving the BLEU score of CodeT5 by up to 55%, boosting Llama 3's BLEU score by 102%, and substantially surpassing all baselines.