A Survey of Code Review Benchmarks and Evaluation Practices in Pre-LLM and LLM Era

TL;DR

This survey comprehensively analyzes 99 code review benchmarks from 2015 to 2025, highlighting trends, limitations, and future directions to improve evaluation practices in both pre-LLM and LLM eras.

Contribution

It provides a systematic taxonomy of code review research, analyzes existing benchmarks, and outlines future directions for more effective evaluation of LLM-based code review tools.

Findings

Shift towards end-to-end generative peer review

Increase in multilingual code review coverage

Decline in standalone change understanding tasks

Abstract

Code review is a critical practice in modern software engineering, helping developers detect defects early, improve code quality, and facilitate knowledge sharing. With the rapid advancement of large language models (LLMs), a growing body of work has explored automated support for code review. However, progress in this area is hindered by the lack of a systematic understanding of existing benchmarks and evaluation practices. Current code review datasets are scattered, vary widely in design, and provide limited insight into what review capabilities are actually being assessed. In this paper, we present a comprehensive survey of code review benchmarks spanning both the Pre-LLM and LLM eras (2015--2025). We analyze 99 research papers (58 Pre-LLM era and 41 LLM era) and extract key metadata, including datasets, evaluation metrics, data sources, and target tasks. Based on this analysis, we propose a multi-level taxonomy that organizes code review research into five domains and 18 fine-grained tasks. Our study reveals a clear shift toward end-to-end generative peer review, increasing multilingual coverage, and a decline in standalone change understanding tasks. We further identify limitations of current benchmarks and outline future directions, including broader task coverage, dynamic runtime evaluation, and taxonomy-guided fine-grained assessment. This survey provides a structured foundation for developing more realistic and comprehensive benchmarks for LLM-based code review.