Context Conquers Parameters: Outperforming Proprietary LLM in Commit Message Generation

TL;DR

This paper demonstrates that open-source LLMs, when properly refined, can generate commit messages that match or surpass proprietary models like GPT-4, addressing privacy and sustainability concerns in software development.

Contribution

The study introduces OMEGA, a novel approach using a 4-bit quantized open-source LLM for commit message generation, achieving state-of-the-art results.

Findings

Open-source LLMs can produce comparable commit messages to proprietary models.

Contextual refinements significantly improve open-source LLM performance.

OMEGA surpasses GPT-4 in practitioners' preference for commit messages.

Abstract

Commit messages provide descriptions of the modifications made in a commit using natural language, making them crucial for software maintenance and evolution. Recent developments in Large Language Models (LLMs) have led to their use in generating high-quality commit messages, such as the Omniscient Message Generator (OMG). This method employs GPT-4 to produce state-of-the-art commit messages. However, the use of proprietary LLMs like GPT-4 in coding tasks raises privacy and sustainability concerns, which may hinder their industrial adoption. Considering that open-source LLMs have achieved competitive performance in developer tasks such as compiler validation, this study investigates whether they can be used to generate commit messages that are comparable with OMG. Our experiments show that an open-source LLM can generate commit messages that are comparable to those produced by OMG. In addition, through a series of contextual refinements, we propose lOcal MessagE GenerAtor (OMEGA) , a CMG approach that uses a 4-bit quantized 8B open-source LLM. OMEGA produces state-of-the-art commit messages, surpassing the performance of GPT-4 in practitioners' preference.