Generating Code with the Help of Retrieved Template Functions and Stack Overflow Answers

TL;DR

This paper introduces a retrieval-augmented code generation framework that enhances sequence-to-sequence models with relevant code snippets and Stack Overflow data, leading to significant improvements in code autocompletion metrics.

Contribution

The authors propose a novel retrieval-guided code generator utilizing multiple retrieval models and datasets, achieving state-of-the-art results on CodeSearchNet and creating a new dataset for code and Stack Overflow alignment.

Findings

4% reduction in cross-entropy loss

15% improvement in edit distance

44% increase in BLEU score

Abstract

We approach the important challenge of code autocompletion as an open-domain task, in which a sequence-to-sequence code generator model is enhanced with the ability to attend to reference code snippets supplied by a semantic code search engine. In this work, we present a novel framework to precisely retrieve template functions as well as intent-snippet pairs and effectively train such a retrieval-guided code generator. To demonstrate the effectiveness of our model designs, we perform extensive experiments with CodeSearchNet which contains template functions and CoNaLa which contains Stack Overflow intent-snippet pairs. We also investigate different retrieval models, including Elasticsearch, DPR, and our fusion representation search model, which currently holds the number one spot on the CodeSearchNet leaderboard. We observe improvements by leveraging multiple database elements and further gain from retrieving diverse data points by using Maximal Marginal Relevance. Overall, we see a 4% improvement to cross-entropy loss, a 15% improvement to edit distance, and a 44% improvement to BLEU score when retrieving template functions. We see subtler improvements of 2%, 11%, and 6% respectively when retrieving Stack Overflow intent-snippet pairs. We also create a novel Stack Overflow-Function Alignment dataset, which consists of 150K tuples of functions and Stack Overflow intent-snippet pairs that are of help in writing the associated function, of which 1.7K are manually curated.