An LLM-as-Judge Metric for Bridging the Gap with Human Evaluation in SE Tasks

TL;DR

SE-Jury is a novel ensemble-based evaluation metric for software artifacts generated by LLMs, significantly improving correlation with human judgment across multiple SE tasks, thus offering a scalable alternative to manual assessment.

Contribution

This paper introduces SE-Jury, the first LLM-based ensemble evaluation metric tailored for assessing software artifact correctness, combining multiple strategies for improved accuracy.

Findings

SE-Jury outperforms existing metrics with 29.6% to 140.8% higher correlation to human judgment.

It achieves agreement levels with human annotators close to inter-annotator agreement.

Demonstrates effectiveness across code generation, repair, and summarization tasks.

Abstract

Large Language Models (LLMs) and other automated techniques have been increasingly used to support software developers by generating software artifacts such as code snippets, patches, and comments. However, accurately assessing the correctness of these generated artifacts remains a significant challenge. On one hand, human evaluation provides high accuracy but is labor-intensive and lacks scalability. On the other hand, many automatic evaluation metrics are scalable and require minimal human effort, but they often fail to accurately reflect the actual correctness of generated software artifacts. In this paper, we present SE-Jury, the first evaluation metric for LLM-as-Ensemble-Judge specifically designed to accurately assess the correctness of generated software artifacts. SE-Jury first defines five distinct evaluation strategies, each implemented by an independent judge. A dynamic team selection mechanism then identifies the most appropriate subset of judges as a team to produce a final correctness score through ensembling. We evaluate SE-Jury across a diverse set of software engineering (SE) benchmarks that span three popular SE tasks: code generation, automated program repair, and code summarization. Results demonstrate that SE-Jury consistently achieves a higher correlation with human judgments, with improvements ranging from 29.6% to 140.8% over existing automatic metrics. SE-Jury reaches agreement levels with human annotators that are close to inter-annotator agreement in code generation and program repair. These findings underscore SE-Jury's potential as a scalable and reliable alternative to human evaluation in these SE tasks.