What Builds Effective In-Context Examples for Code Generation?

TL;DR

This paper systematically examines how different features of in-context code examples influence large language models' code generation, highlighting the importance of variable naming and identifying current limitations in reflection capabilities.

Contribution

It provides a controlled analysis of code features affecting ICL effectiveness, revealing the critical role of naming and the challenges in reflection-based learning.

Findings

Proper variable and function naming significantly improves performance.

LLMs prioritize semantic meaning over formatting conventions.

Current models struggle with extracting generalizable insights from code.

Abstract

In-Context Learning (ICL) has emerged as a promising solution to enhance the code generation capabilities of Large Language Models (LLMs), which incorporates code examples inside the prompt to let LLMs learn from demonstrations. However, despite the substantial effectiveness of the code example-based ICL approach, the specific features (e.g., identifier naming styles, code formatting, solution insight) within the ICL-provided code examples that significantly contribute to the ICL's effectiveness remain unclear. This paper systematically investigates the impact of various code features on ICL with code examples through controlled ablation studies. Our findings reveal that the appropriate naming of variables and functions is crucial for effective code generation, with their elimination leading to performance decreases of up to 30 percentage points. We further demonstrate that LLMs prioritize semantically meaningful identifier names over formatting conventions, with language-specific preferences regarding identifier verbosity. Additionally, our investigation into ICL's potential for enhancing reflection and inference capabilities reveals that current LLMs struggle to extract generalizable problem-solving insights from similar code solutions, despite being capable of utilizing direct information effectively. These findings are expected to provide valuable insights for optimizing ICL systems in code generation applications and highlight fundamental challenges in reflection-based learning for code generation tasks.