RAPGen: An Approach for Fixing Code Inefficiencies in Zero-Shot

TL;DR

RAPGen is a novel method that leverages retrieval-augmented prompt generation with large language models to effectively fix performance bugs in code, outperforming existing methods in accuracy and similarity to developer fixes.

Contribution

This work introduces RAPGen, a retrieval-augmented prompt generation approach that improves zero-shot performance bug fixing by combining knowledge-base retrieval with large language models.

Findings

Achieves ~60% success rate in generating performance improvements comparable to developers.

Verbatim fixes are produced in ~42% of cases.

Outperforms various prompt and state-of-the-art methods in performance bug fixing.

Abstract

Performance bugs are non-functional bugs that can even manifest in well-tested commercial products. Fixing these performance bugs is an important yet challenging problem. In this work, we address this challenge and present a new approach called Retrieval-Augmented Prompt Generation (RAPGen). Given a code snippet with a performance issue, RAPGen first retrieves a prompt instruction from a pre-constructed knowledge-base of previous performance bug fixes and then generates a prompt using the retrieved instruction. It then uses this prompt on a Large Language Model (such as Codex) in zero-shot to generate a fix. We compare our approach with the various prompt variations and state of the art methods in the task of performance bug fixing. Our evaluation shows that RAPGen can generate performance improvement suggestions equivalent or better than a developer in ~60% of the cases, getting ~42% of them verbatim, in an expert-verified dataset of past performance changes made by C# developers.