FastCoder: Accelerating Repository-level Code Generation via Efficient Retrieval and Verification

TL;DR

FastCoder is a specialized inference acceleration method for repository-level code generation that leverages retrieval and caching to significantly improve speed without sacrificing code quality.

Contribution

FastCoder introduces a retrieval-based acceleration approach tailored for code generation, combining multi-source datastores, controlled retrieval timing, and cache strategies.

Findings

Achieves up to 2.54x speedup in code generation tasks.

Outperforms state-of-the-art inference acceleration methods by up to 88%.

Can be integrated with existing approaches for over 2.6x speedup.

Abstract

Code generation is a latency-sensitive task that demands high timeliness. However, with the growing interest and inherent difficulty in repository-level code generation, most existing code generation studies focus on improving the correctness of generated code while overlooking the inference efficiency, which is substantially affected by the overhead during LLM generation. Although there has been work on accelerating LLM inference, these approaches are not tailored to the specific characteristics of code generation; instead, they treat code the same as natural language sequences and ignore its unique syntax and semantic characteristics, which are also crucial for improving efficiency. Consequently, these approaches exhibit limited effectiveness in code generation tasks, particularly for repository-level scenarios with considerable complexity and difficulty. To alleviate this issue, following draft-verification paradigm, we propose FastCoder, a simple yet highly efficient inference acceleration approach specifically designed for code generation, without compromising the quality of the output. FastCoder constructs a multi-source datastore, providing access to both general and project-specific knowledge, facilitating the retrieval of high-quality draft sequences. Moreover, FastCoder reduces the retrieval cost by controlling retrieval timing, and enhances efficiency through parallel retrieval and a context- and LLM preference-aware cache. Experimental results show that FastCoder can reach up to 2.53x and 2.54x speedup compared to autoregressive decoding in repository-level and standalone code generation tasks, respectively, outperforming state-of-the-art inference acceleration approaches by up to 88%. FastCoder can also be integrated with existing correctness-focused code generation approaches to accelerate the LLM generation process, and reach a speedup exceeding 2.6x.