RepoMasterEval: Evaluating Code Completion via Real-World Repositories

TL;DR

RepoMasterEval is a new benchmark for evaluating code completion models using real-world repository data, emphasizing practical scenarios and test effectiveness to better reflect real development environments.

Contribution

It introduces a novel benchmark constructed from real repositories, incorporating mutation testing and manual test case creation to improve evaluation accuracy.

Findings

Test argumentation significantly improves model accuracy.

RepoMasterEval reports performance variance in real-world scenarios.

The benchmark correlates well with practical model performance.

Abstract

With the growing reliance on automated code completion tools in software development, the need for comprehensive evaluation benchmarks has become critical. Existing benchmarks focus more on code completion in function and class level by providing text descriptions to prompt the model. By contrast, such descriptive prompt is commonly unavailable in real development and code completion can occur in wider range of situations such as in the middle of a function or a code block. These limitations makes existing evaluation benchmarks poorly align with the practical scenarios of code completion tools. In this paper, we propose RepoMasterEval, a novel benchmark for evaluating code completion models constructed from real-world repositories. Each benchmark datum is generated by masking a code snippet (ground truth) from one source code file with existing test suites. To improve test accuracy of model generated code, we employ mutation testing to measure the effectiveness of the test cases and we manually crafted new test cases for those test suites with low mutation score. Our empirical evaluation on 10 state-of-the-art models shows that test argumentation is critical in improving the accuracy of the benchmark and RepoMasterEval is able to report variance in model performance in real-world scenarios. The deployment of RepoMasterEval also revealed that the benchmark is useful to give accurate feedback during model training and the score is in high correlation with the model's performance in practice.