HumanEvo: An Evolution-aware Benchmark for More Realistic Evaluation of Repository-level Code Generation

TL;DR

This paper introduces HumanEvo, an evolution-aware benchmark for evaluating repository-level code generation by LLMs, addressing the limitations of previous methods that ignore software evolution over time.

Contribution

It constructs an evolution-aware dataset, categorizes it by dependency levels, and evaluates seven LLMs, revealing overestimations in performance by previous evaluation methods.

Findings

Previous methods overestimate LLM performance by 10-61%.

Evolution-aware evaluation provides more accurate performance insights.

The benchmark and toolbox facilitate future research in realistic code generation evaluation.

Abstract

To evaluate the repository-level code generation capabilities of Large Language Models (LLMs) in complex real-world software development scenarios, many evaluation methods have been developed. These methods typically leverage contextual code from the latest version of a project to assist LLMs in accurately generating the desired function. However, such evaluation methods fail to consider the dynamic evolution of software projects over time, which we refer to as evolution-ignored settings. This in turn results in inaccurate evaluation of LLMs' performance. In this paper, we conduct an empirical study to deeply understand LLMs' code generation performance within settings that reflect the evolution nature of software development. To achieve this, we first construct an evolution-aware repository-level code generation dataset, namely HumanEvo, equipped with an automated execution-based evaluation tool. Second, we manually categorize HumanEvo according to dependency levels to more comprehensively analyze the model's performance in generating functions with different dependency levels. Third, we conduct extensive experiments on HumanEvo with seven representative and diverse LLMs to verify the effectiveness of the proposed benchmark. We obtain several important findings through our experimental study. For example, we find that previous evolution-ignored evaluation methods result in inflated performance of LLMs, with performance overestimations ranging from 10.0% to 61.1% under different context acquisition methods, compared to the evolution-aware evaluation approach. Based on the findings, we give actionable suggestions for more realistic evaluation of LLMs on code generation. We also build a shared evolution-aware code generation toolbox to facilitate future research.