The First Prompt Counts the Most! An Evaluation of Large Language Models on Iterative Example-Based Code Generation

TL;DR

This study evaluates how large language models perform in code generation based on iterative input-output examples, revealing significant challenges and opportunities for improving example-based prompting strategies.

Contribution

It is the first comprehensive analysis of LLMs in example-based code generation, introducing an iterative evaluation framework and a new benchmark of 172 functionalities.

Findings

LLMs' performance drops over 60% with example-based prompts compared to natural language.

Most functionalities are correctly implemented in the first iteration.

Combining I/O examples with natural language improves LLM accuracy.

Abstract

The capabilities of Large Language Models (LLMs) in code generation have been extensively studied, particularly for implementing target functionalities from natural-language descriptions. Alternatively, input-output (I/O) examples provide an accessible, unambiguous, and flexible way to describe functionalities. However, their inherent diversity, opaqueness, and incompleteness impose greater challenges for understanding and implementing the target requirements. Therefore, generating code from I/O examples (i.e., example-based code generation) provides a new perspective, allowing us to additionally evaluate LLMs' capability to infer target functionalities from limited information and to process new-form requirements. However, related research about LLMs in example-based code generation remains largely unexplored. To fill this gap, this paper presents the first comprehensive study on example-based code generation using LLMs. We adopt an iterative evaluation framework and formalize the objective of example-based code generation as two sequential sub-objectives: generating code conforming to the given examples and generating code that successfully implements the target functionalities from (iteratively) given examples. We assess six state-of-the-art LLMs using a new benchmark of 172 diverse target functionalities. The results demonstrate that when requirements are described using iterative I/O examples rather than natural language, the LLMs' score decreases by over 60%, and the vast majority (even over 95%) of successfully implemented functionalities are achieved in the first round of the iterations. Furthermore, we also find that combining I/O examples with even imprecise and fragmental natural language descriptions greatly improves LLM performance, and the selection of initial I/O examples can also influence the score, suggesting opportunities for prompt optimization.