Evaluating the Effectiveness and Efficiency of Demonstration Retrievers in RAG for Coding Tasks

TL;DR

This paper systematically evaluates the trade-offs between effectiveness and efficiency of various retrievers in Retrieval-Augmented Generation for coding tasks, highlighting the strengths and limitations of each method.

Contribution

It provides a comprehensive comparison of six retrievers, analyzing their performance and efficiency trade-offs across multiple coding tasks, which was previously underexplored.

Findings

BM25 is highly effective but less efficient at large scale.

Approximate dense retrievers significantly improve speed with minimal accuracy loss.

Increasing demonstrations does not always enhance effectiveness and can increase latency.

Abstract

Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by integrating external knowledge bases, achieving state-of-the-art results in various coding tasks. The core of RAG is retrieving demonstration examples, which is essential to balance effectiveness (generation quality) and efficiency (retrieval time) for optimal performance. However, the high-dimensional nature of code representations and large knowledge bases often create efficiency bottlenecks, which are overlooked in previous research. This paper systematically evaluates the efficiency-effectiveness trade-off of retrievers across three coding tasks: Program Synthesis, Commit Message Generation, and Assertion Generation. We examined six retrievers: two sparse (BM25 and BM25L) and four dense retrievers, including one exhaustive dense retriever (SBERT's Semantic Search) and three approximate dense retrievers (ANNOY, LSH, and HNSW). Our findings show that while BM25 excels in effectiveness, it suffers in efficiency as the knowledge base grows beyond 1000 entries. In large-scale retrieval, efficiency differences become more pronounced, with approximate dense retrievers offering the greatest gains. For instance, in Commit Generation task, HNSW achieves a 44x speed up, while only with a 1.74% drop in RougeL compared with BM25. Our results also show that increasing the number of demonstrations in the prompt doesn't always improve the effectiveness and can increase latency and lead to incorrect outputs. Our findings provide valuable insights for practitioners aiming to build efficient and effective RAG systems for coding tasks.