What Makes Large Language Models Reason in (Multi-Turn) Code Generation?

TL;DR

This paper systematically investigates prompting strategies, especially multi-turn re-prompting, to enhance large language models' reasoning in code generation, demonstrating improvements through both prompting and finetuning on competitive programming benchmarks.

Contribution

It provides a comprehensive analysis of prompting techniques for multi-turn code reasoning and shows how finetuning with these strategies improves model performance and scalability.

Findings

Certain prompting strategies consistently improve performance across models.

Finetuning with optimal prompts internalizes reasoning, boosting multi-turn code generation.

Strategies are effective for both small and large sampling budgets.

Abstract

Prompting techniques such as chain-of-thought have established themselves as a popular vehicle for improving the outputs of large language models (LLMs). For code generation, however, their exact mechanics and efficacy are under-explored. We thus investigate the effects of a wide range of prompting strategies with a focus on automatic re-prompting over multiple turns and computational requirements. After systematically decomposing reasoning, instruction, and execution feedback prompts, we conduct an extensive grid search on the competitive programming benchmarks CodeContests and TACO for multiple LLM families and sizes (Llama 3.0 and 3.1, 8B, 70B, 405B, and GPT-4o). Our study reveals strategies that consistently improve performance across all models with small and large sampling budgets. We then show how finetuning with such an optimal configuration allows models to internalize the induced reasoning process and obtain improvements in performance and scalability for multi-turn code generation.