ExecRepoBench: Multi-level Executable Code Completion Evaluation

TL;DR

This paper introduces ExecRepoBench, a new multi-level benchmark for evaluating code completion in complex, multi-file Python projects, and demonstrates a fine-tuned open-source model that outperforms existing benchmarks.

Contribution

It presents a novel multi-level, repository-level benchmark and a grammar-based code completion methodology, along with a fine-tuned open-source model, improving real-world code completion performance.

Findings

Qwen2.5-Coder-Instruct-C outperforms prior baselines across languages.

ExecRepoBench provides 1.2K real-world Python samples.

The framework enables more realistic code completion evaluation.

Abstract

Code completion has become an essential tool for daily software development. Existing evaluation benchmarks often employ static methods that do not fully capture the dynamic nature of real-world coding environments and face significant challenges, including limited context length, reliance on superficial evaluation metrics, and potential overfitting to training datasets. In this work, we introduce a novel framework for enhancing code completion in software development through the creation of a repository-level benchmark ExecRepoBench and the instruction corpora Repo-Instruct, aim at improving the functionality of open-source large language models (LLMs) in real-world coding scenarios that involve complex interdependencies across multiple files. ExecRepoBench includes 1.2K samples from active Python repositories. Plus, we present a multi-level grammar-based completion methodology conditioned on the abstract syntax tree to mask code fragments at various logical units (e.g. statements, expressions, and functions). Then, we fine-tune the open-source LLM with 7B parameters on Repo-Instruct to produce a strong code completion baseline model Qwen2.5-Coder-Instruct-C based on the open-source model. Qwen2.5-Coder-Instruct-C is rigorously evaluated against existing benchmarks, including MultiPL-E and ExecRepoBench, which consistently outperforms prior baselines across all programming languages. The deployment of \ourmethod{} can be used as a high-performance, local service for programming development\footnote{\url{https://execrepobench.github.io/}}.