RAG or Fine-tuning? A Comparative Study on LCMs-based Code Completion in Industry

TL;DR

This study compares retrieval-augmented generation and fine-tuning for industrial code completion using large code models, revealing RAG's superior accuracy, efficiency, and scalability in real-world scenarios.

Contribution

It provides the first comprehensive comparison of RAG and fine-tuning approaches for industrial code completion, including practical insights from large-scale industrial data.

Findings

RAG with effective embedding models outperforms fine-tuning in accuracy.

BM25 retrieval method offers the best effectiveness and efficiency among RAG techniques.

Combining RAG and fine-tuning yields further improvements and RAG scales better with larger codebases.

Abstract

Code completion, a crucial practice in industrial settings, helps developers improve programming efficiency by automatically suggesting code snippets during development. With the emergence of Large Code Models (LCMs), this field has witnessed significant advancements. Due to the natural differences between open-source and industrial codebases, such as coding patterns and unique internal dependencies, it is a common practice for developers to conduct domain adaptation when adopting LCMs in industry. There exist multiple adaptation approaches, among which retrieval-augmented generation (RAG) and fine-tuning are the two most popular paradigms. However, no prior research has explored the trade-off of the two approaches in industrial scenarios. To mitigate the gap, we comprehensively compare the two paradigms including Retrieval-Augmented Generation (RAG) and Fine-tuning (FT), for industrial code completion in this paper. In collaboration with Tencent's WXG department, we collect over 160,000 internal C++ files as our codebase. We then compare the two types of adaptation approaches from three dimensions that are concerned by industrial practitioners, including effectiveness, efficiency, and parameter sensitivity, using six LCMs. Our findings reveal that RAG, when implemented with appropriate embedding models that map code snippets into dense vector representations, can achieve higher accuracy than fine-tuning alone. Specifically, BM25 presents superior retrieval effectiveness and efficiency among studied RAG methods. Moreover, RAG and fine-tuning are orthogonal and their combination leads to further improvement. We also observe that RAG demonstrates better scalability than FT, showing more sustained performance gains with larger scales of codebase.