In-Context Learning as an Effective Estimator of Functional Correctness of LLM-Generated Code

TL;DR

This paper introduces an in-context learning method to estimate the functional correctness of LLM-generated code, improving the accuracy of quality predictions without requiring test cases.

Contribution

It proposes a novel ICL-based approach for code quality estimation that enhances existing zero-shot and QPP methods in software development workflows.

Findings

Few-shot examples improve estimation accuracy

ICL enhances existing QPP approaches

Zero-shot approach benefits from ICL

Abstract

When applying LLM-based code generation to software development projects that follow a feature-driven or rapid application development approach, it becomes necessary to estimate the functional correctness of the generated code in the absence of test cases. Just as a user selects a relevant document from a ranked list of retrieved ones, a software generation workflow requires a developer to choose (and potentially refine) a generated solution from a ranked list of alternative solutions, ordered by their posterior likelihoods. This implies that estimating the quality of a ranked list -- akin to estimating "relevance" for query performance prediction (QPP) in IR -- is also crucial for generative software development, where quality is defined in terms of "functional correctness". In this paper, we propose an in-context learning (ICL) based approach for code quality estimation. Our findings demonstrate that providing few-shot examples of functionally correct code from a training set enhances the performance of existing QPP approaches as well as a zero-shot-based approach for code quality estimation.