SynthCoder: A Synthetical Strategy to Tune LLMs for Code Completion

TL;DR

SynthCoder is a novel approach that combines diverse data augmentation, contextual enrichment, and a two-stage training process to significantly improve code completion performance of LLMs, especially in repository-level scenarios.

Contribution

The paper introduces SynthCoder, integrating industry practices and a two-stage training method to achieve state-of-the-art code completion results and address common model issues.

Findings

Outperforms existing models on multiple repository-level benchmarks.

Effectively mitigates code repetition issues in LLMs.

Enhances code completion accuracy with enriched contextual data.

Abstract

Code completion is a prominent application of Large Language Models (LLMs) in software engineering. Due to the near real-time response requirements of this task, base models with small to medium-sized parameters are typically employed, supplemented by various optimization and post-training techniques. However, these optimization methods often have trade-offs, leading to a seesaw effect where performance improvements on certain datasets or metrics are accompanied by degradations on others -- sometimes even falling below the baseline model's performance. This paper proposes SynthCoder, a model that integrates leading industry practices to achieve state-of-the-art performance on the Fill-in-the-Middle (FIM) code completion task. In specific, we first construct a diverse dataset by combining Abstract Syntax Tree (AST) node extraction with heuristics that simulate developer behavior. Then we enrich our training corpus with cross-file contextual information using the BM25 algorithm and call graphs, enhancing the model's ability to perform code completion in both file-level and repository-level scenarios. As the last step, we employ a two-stage training process using the Seed-Coder-8B-Base as the base model. First, we fine-tune the model using Curriculum Learning technology. Following this, we perform alignment using Direct Preference Optimization (DPO) with preference pairs generated through Rejection Sampling. Experimental results demonstrate that our final model excels on mainstream repository-level code completion benchmarks, including aiXcoder, ExecRepoBench, CrossCodeEval, and CoLT. Furthermore, our carefully curated training set effectively mitigates the model's tendency to just repeat existing code, a common issue existing in various code completion models.