Evaluating Generated Commit Messages with Large Language Models

TL;DR

This paper explores using Large Language Models as automated evaluators for generated commit messages, showing they outperform traditional metrics and approach human-level assessment in quality evaluation.

Contribution

It introduces a novel LLM-based evaluation method for commit messages, demonstrating its effectiveness and robustness compared to traditional automatic metrics.

Findings

LLMs with Chain-of-Thought reasoning excel in commit message evaluation

The proposed LLM evaluator outperforms traditional metrics in accuracy

The method offers a scalable alternative to human evaluation

Abstract

Commit messages are essential in software development as they serve to document and explain code changes. Yet, their quality often falls short in practice, with studies showing significant proportions of empty or inadequate messages. While automated commit message generation has advanced significantly, particularly with Large Language Models (LLMs), the evaluation of generated messages remains challenging. Traditional reference-based automatic metrics like BLEU, ROUGE-L, and METEOR have notable limitations in assessing commit message quality, as they assume a one-to-one mapping between code changes and commit messages, leading researchers to rely on resource-intensive human evaluation. This study investigates the potential of LLMs as automated evaluators for commit message quality. Through systematic experimentation with various prompt strategies and state-of-the-art LLMs, we demonstrate that LLMs combining Chain-of-Thought reasoning with few-shot demonstrations achieve near human-level evaluation proficiency. Our LLM-based evaluator significantly outperforms traditional metrics while maintaining acceptable reproducibility, robustness, and fairness levels despite some inherent variability. This work conducts a comprehensive preliminary study on using LLMs for commit message evaluation, offering a scalable alternative to human assessment while maintaining high-quality evaluation.