Contextual Code Retrieval for Commit Message Generation: A Preliminary Study

TL;DR

This paper introduces C3Gen, a method that improves commit message generation by retrieving and incorporating relevant code snippets to provide richer context, leading to more informative and practical commit messages.

Contribution

It proposes a novel contextual code retrieval approach for commit message generation, enhancing existing models with repository-wide code context for better quality.

Findings

C3Gen improves the informativeness of generated commit messages.

Incorporating contextual code enhances model performance across multiple metrics.

Human evaluation shows increased developer satisfaction with C3Gen outputs.

Abstract

A commit message describes the main code changes in a commit and plays a crucial role in software maintenance. Existing commit message generation (CMG) approaches typically frame it as a direct mapping which inputs a code diff and produces a brief descriptive sentence as output. However, we argue that relying solely on the code diff is insufficient, as raw code diff fails to capture the full context needed for generating high-quality and informative commit messages. In this paper, we propose a contextual code retrieval-based method called C3Gen to enhance CMG by retrieving commit-relevant code snippets from the repository and incorporating them into the model input to provide richer contextual information at the repository scope. In the experiments, we evaluated the effectiveness of C3Gen across various models using four objective and three subjective metrics. Meanwhile, we design and conduct a human evaluation to investigate how C3Gen-generated commit messages are perceived by human developers. The results show that by incorporating contextual code into the input, C3Gen enables models to effectively leverage additional information to generate more comprehensive and informative commit messages with greater practical value in real-world development scenarios. Further analysis underscores concerns about the reliability of similaritybased metrics and provides empirical insights for CMG.