Assessing the Answerability of Queries in Retrieval-Augmented Code Generation

TL;DR

This paper introduces a new task and benchmark dataset for evaluating whether retrieval-augmented code generation models can produce answerable, correct code based on user queries and retrieved APIs, highlighting the task's difficulty.

Contribution

It proposes the answerability evaluation task and creates the RaCGEval benchmark dataset, providing a new way to assess and improve retrieval-augmented code generation models.

Findings

Baseline models achieve only 46.7% performance on answerability.

Answerability remains a very challenging task for current models.

The paper discusses potential methods to significantly improve model performance.

Abstract

Thanks to unprecedented language understanding and generation capabilities of large language model (LLM), Retrieval-augmented Code Generation (RaCG) has recently been widely utilized among software developers. While this has increased productivity, there are still frequent instances of incorrect codes being provided. In particular, there are cases where plausible yet incorrect codes are generated for queries from users that cannot be answered with the given queries and API descriptions. This study proposes a task for evaluating answerability, which assesses whether valid answers can be generated based on users' queries and retrieved APIs in RaCG. Additionally, we build a benchmark dataset called Retrieval-augmented Code Generability Evaluation (RaCGEval) to evaluate the performance of models performing this task. Experimental results show that this task remains at a very challenging level, with baseline models exhibiting a low performance of 46.7%. Furthermore, this study discusses methods that could significantly improve performance.