Evaluating and Achieving Controllable Code Completion in Code LLM

TL;DR

This paper introduces a new benchmark for instruction-guided code completion, evaluates numerous LLMs, and develops a fine-tuned model that outperforms existing ones, highlighting gaps and future directions in code LLMs.

Contribution

It presents the first instruction-guided code completion benchmark, a data synthesis pipeline for fine-tuning, and a new model achieving state-of-the-art results.

Findings

Open-source models lag behind proprietary ones in instruction-following.

The new benchmark reveals significant gaps in instruction-following capabilities.

Fine-tuning with synthesized data improves code completion performance.

Abstract

Code completion has become a central task, gaining significant attention with the rise of large language model (LLM)-based tools in software engineering. Although recent advances have greatly improved LLMs' code completion abilities, evaluation methods have not advanced equally. Most current benchmarks focus solely on functional correctness of code completions based on given context, overlooking models' ability to follow user instructions during completion-a common scenario in LLM-assisted programming. To address this limitation, we present the first instruction-guided code completion benchmark, Controllable Code Completion Benchmark (C3-Bench), comprising 2,195 carefully designed completion tasks. Through comprehensive evaluation of over 40 mainstream LLMs across C3-Bench and conventional benchmarks, we reveal substantial gaps in instruction-following capabilities between open-source and advanced proprietary models during code completion tasks. Moreover, we develop a straightforward data synthesis pipeline that leverages Qwen2.5-Coder to generate high-quality instruction-completion pairs for supervised fine-tuning (SFT). The resulting model, Qwen2.5-Coder-C3, achieves state-of-the-art performance on C3-Bench. Our findings provide valuable insights for enhancing LLMs' code completion and instruction-following capabilities, establishing new directions for future research in code LLMs. To facilitate reproducibility and foster further research in code LLMs, we open-source all code, datasets, and models.