ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code

TL;DR

ComplexCodeEval is a comprehensive benchmark that evaluates large code models across diverse real-world programming tasks using extensive, annotated datasets to better reflect practical development challenges.

Contribution

The paper introduces ComplexCodeEval, a new benchmark with diverse tasks and datasets, addressing limitations of prior narrow evaluation methods for large code models.

Findings

Context enhances model performance.

Data leakage can cause overestimation.

Evaluation accuracy is crucial for progress.

Abstract

In recent years, the application of large language models (LLMs) to code-related tasks has gained significant attention. However, existing evaluation benchmarks often focus on limited scenarios, such as code generation or completion, which do not reflect the diverse challenges developers face in real-world contexts. To address this, we introduce ComplexCodeEval, a benchmark designed to assess LCMs in various development tasks, including code generation, completion, API recommendation, and test case generation. It includes 3,897 Java samples and 7,184 Python samples from high-star GitHub repositories, each annotated with function signatures, docstrings, and API references to simulate real development environments. Our experiments across ten LCMs reveal that context improves performance and that data leakage can lead to overestimation, highlighting the need for more accurate evaluations.