Repoformer: Selective Retrieval for Repository-Level Code Completion

TL;DR

This paper introduces a selective retrieval framework for repository-level code completion that improves efficiency and robustness by enabling code models to decide when retrieval is beneficial, achieving state-of-the-art results.

Contribution

The paper presents a self-supervised learning approach allowing code models to selectively perform retrieval, enhancing performance and speed in repository-level code completion tasks.

Findings

Achieves state-of-the-art performance on multiple benchmarks.

Provides up to 70% inference speedup without performance loss.

Effectively handles different models, retrievers, and languages.

Abstract

Recent advances in retrieval-augmented generation (RAG) have initiated a new era in repository-level code completion. However, the invariable use of retrieval in existing methods exposes issues in both efficiency and robustness, with a large proportion of the retrieved contexts proving unhelpful or harmful to code language models (code LMs). In this paper, we propose a selective RAG framework to avoid retrieval when unnecessary. To power this framework, we design a self-supervised learning approach to enable a code LM to accurately self-evaluate whether retrieval can improve its output quality and robustly leverage the potentially noisy retrieved contexts. Using this LM as both the selective RAG policy and the generation model, our framework achieves state-of-the-art repository-level code completion performance on diverse benchmarks including RepoEval, CrossCodeEval, and CrossCodeLongEval, a new long-form code completion benchmark. Meanwhile, our analyses show that selectively retrieving brings as much as 70% inference speedup in the online serving setting without harming the performance. We further demonstrate that our framework is able to accommodate different generation models, retrievers, and programming languages. These advancements position our framework as an important step towards more accurate and efficient repository-level code completion.