DeepCRCEval: Revisiting the Evaluation of Code Review Comment Generation

TL;DR

This paper introduces DeepCRCEval, a new evaluation framework for code review comments that combines human and LLM assessments, addressing limitations of traditional text similarity metrics and improving evaluation reliability and efficiency.

Contribution

It proposes an innovative evaluation framework integrating human and LLM evaluators, and introduces LLM-Reviewer as a new baseline for comment quality assessment.

Findings

Less than 10% of benchmark comments are high quality for automation.

DeepCRCEval effectively distinguishes comment quality levels.

Incorporating LLM evaluators reduces evaluation time and cost significantly.

Abstract

Code review is a vital but demanding aspect of software development, generating significant interest in automating review comments. Traditional evaluation methods for these comments, primarily based on text similarity, face two major challenges: inconsistent reliability of human-authored comments in open-source projects and the weak correlation of text similarity with objectives like enhancing code quality and detecting defects. This study empirically analyzes benchmark comments using a novel set of criteria informed by prior research and developer interviews. We then similarly revisit the evaluation of existing methodologies. Our evaluation framework, DeepCRCEval, integrates human evaluators and Large Language Models (LLMs) for a comprehensive reassessment of current techniques based on the criteria set. Besides, we also introduce an innovative and efficient baseline, LLM-Reviewer, leveraging the few-shot learning capabilities of LLMs for a target-oriented comparison. Our research highlights the limitations of text similarity metrics, finding that less than 10% of benchmark comments are high quality for automation. In contrast, DeepCRCEval effectively distinguishes between high and low-quality comments, proving to be a more reliable evaluation mechanism. Incorporating LLM evaluators into DeepCRCEval significantly boosts efficiency, reducing time and cost by 88.78% and 90.32%, respectively. Furthermore, LLM-Reviewer demonstrates significant potential of focusing task real targets in comment generation.