Fixing Large Language Models' Specification Misunderstanding for Better Code Generation

TL;DR

This paper introduces {}FiX, a novel prompting technique that enhances large language models' code generation by improving their understanding of specifications through test case analysis and feedback, leading to significant performance gains.

Contribution

It proposes a new {}FiX prompting method that combines thought-eliciting and feedback-based prompts, and explores their synergy to better align LLMs with programming specifications.

Findings

{}FiX significantly improves code generation accuracy.

It outperforms 15 baselines on six benchmarks.

Effective for models like ChatGPT and DeepSeek-Coder.

Abstract

Code generation is to automatically generate source code conforming to a given programming specification, which has received extensive attention especially with the development of large language models (LLMs). Due to the inherent difficulty of code generation, the code generated by LLMs may not be aligned with the specification. Although thought-eliciting prompting techniques have been proposed to enhance the code generation performance of LLMs, producing correct understanding for complicated programming problems remains challenging, resulting in unsatisfactory performance. Also, some feedback-based prompting techniques have been proposed to fix incorrect code using error messages produced by test execution. However, when the generated code deviates significantly from the ground truth, they encounter difficulties in improving performance based on such coarse-grained information. In this work, we propose a novel prompting technique, called {\mu}FiX, to improve the code generation performance of LLMs by devising both sophisticated thought-eliciting prompting and feedback-based prompting and making the first exploration on their synergy. It first exploits test case analysis to obtain specification understanding and enables a self-improvement process to identify and refine the misunderstanding in the thought-eliciting prompting phase. {\mu}FiX further fixes the specification understanding towards the direction reducing the gap between the provided understanding (from the first phase) and the actual understanding implicitly utilized by LLMs for code generation in the feedback-based prompting phase. By improving the understanding with {\mu}FiX, the code generation performance of LLMs can be largely improved. Our evaluation on two advanced LLMs (ChatGPT and DeepSeek-Coder) with six widely-used benchmarks by comparing with 15 baselines, demonstrates the effectiveness of {\mu}FiX ...