Demo-Craft: Using In-Context Learning to Improve Code Generation in Large Language Models

TL;DR

Demo-Craft enhances code generation in large language models by combining in-context learning, demonstration selection, and latent concept learning, resulting in significant improvements in correctness and similarity metrics.

Contribution

The paper introduces Demo-Craft, a novel system that integrates latent concept learning with in-context demonstration selection to improve code generation quality.

Findings

Approximately 2x increase in pass@k metric.

Nearly 3x improvement in correctness@k and similarity@k metrics.

Effective on MBPP and Humaneval datasets.

Abstract

Generating executable code from natural language instructions using Large Language Models (LLMs) poses challenges such as semantic ambiguity and understanding taskspecific contexts. To address these issues, we propose a system called DemoCraft, which enhances code generation by leveraging in-context learning and demonstration selection, combined with latent concept learning. Latent concept learning introduces additional concept tokens, which are trainable embeddings that capture task-specific knowledge. We then test our system on two major datasets: MBPP and Humaneval. Our experimental results demonstrate that the proposed system achieves an approximate 2x increase in the pass@k metric compared to baseline models. Furthermore, we introduce two novel evaluation metrics: correctness@k and similarity@k. Our empirical studies indicate that our system attains nearly a 3x improvement in these metrics as well.