PERC: Plan-As-Query Example Retrieval for Underrepresented Code Generation

TL;DR

PERC introduces a novel retrieval framework that leverages algorithmic plans and pseudocode to improve few-shot code generation, especially for underrepresented programming languages, outperforming existing methods across multiple benchmarks.

Contribution

The paper proposes PERC, a new approach that uses algorithmic plans as queries for example retrieval, enhancing code generation quality in diverse programming languages.

Findings

PERC outperforms state-of-the-art RAG methods on multiple benchmarks.

Using pseudocode effectively captures algorithmic plans across different programming languages.

Retrieving examples based on algorithmic plans improves code generation accuracy.

Abstract

Code generation with large language models has shown significant promise, especially when employing retrieval-augmented generation (RAG) with few-shot examples. However, selecting effective examples that enhance generation quality remains a challenging task, particularly when the target programming language (PL) is underrepresented. In this study, we present two key findings: (1) retrieving examples whose presented algorithmic plans can be referenced for generating the desired behavior significantly improves generation accuracy, and (2) converting code into pseudocode effectively captures such algorithmic plans, enhancing retrieval quality even when the source and the target PLs are different. Based on these findings, we propose Plan-as-query Example Retrieval for few-shot prompting in Code generation (PERC), a novel framework that utilizes algorithmic plans to identify and retrieve effective examples. We validate the effectiveness of PERC through extensive experiments on the CodeContests, HumanEval and MultiPL-E benchmarks: PERC consistently outperforms the state-of-the-art RAG methods in code generation, both when the source and target programming languages match or differ, highlighting its adaptability and robustness in diverse coding environments.