An Empirical Study of Retrieval-Augmented Code Generation: Challenges and Opportunities

TL;DR

This study systematically evaluates retrieval-augmented frameworks for code generation, demonstrating their benefits, analyzing different integration methods, and discussing trade-offs between performance gains and computational costs.

Contribution

It provides a comprehensive empirical analysis of retrieval-augmented code generation, comparing multiple models and integration techniques, and offers practical recommendations for improving performance.

Findings

Retrieval-augmented frameworks improve code generation performance.

BM25 and Sequential Integration Fusion are effective and convenient methods.

Sketch Filling Fusion can further enhance model accuracy.

Abstract

Code generation aims to automatically generate code snippets of specific programming language according to natural language descriptions. The continuous advancements in deep learning, particularly pre-trained models, have empowered the code generation task to achieve remarkable performance. One main challenge of pre-trained models for code generation is the semantic gap between natural language requirements and source code. To address the issue, prior studies typically adopt a retrieval-augmented framework for the task, where the similar code snippets collected by a retrieval process can be leveraged to help understand the requirements and provide guidance for the generation process. However, there is a lack of systematic study on the application of this framework for code generation, including the impact of the final generated results and the specific usage of the framework. In this paper, we choose three popular pre-trained code models, namely CodeGen, UniXcoder, and CodeT5, to assess the impact of the quality and utilization of retrieved code on the retrieval-augmented framework. Our analysis shows that the retrieval-augmented framework is beneficial for improving the performance of the existing pre-trained models. We also provide suggestions on the utilization of the retrieval-augmented code generation framework: BM25 and Sequential Integration Fusion are recommended due to their convenience and superior performance. Sketch Filling Fusion, which extracts a sketch of relevant code, could help the model improve its performance further. Additionally, we conduct experiments to investigate the influence of the retrieval-augmented framework on large language models for code generation, showing the effectiveness of the framework, and we discuss the trade-off between performance improvement and computational costs in each phase within the framework.