Improving ChatGPT Prompt for Code Generation

TL;DR

This paper investigates how prompt design influences ChatGPT's effectiveness in code generation, demonstrating that carefully crafted prompts significantly enhance performance in text-to-code and code-to-code tasks.

Contribution

It introduces prompt optimization strategies, including chain-of-thought techniques, to improve ChatGPT's code generation capabilities based on experimental evaluation.

Findings

Prompt design greatly affects ChatGPT's code generation quality

Chain-of-thought prompts lead to better results

Insights guide future prompt engineering research

Abstract

Automated code generation can be a powerful technique for software development, significantly reducing developers' efforts and time required to create new code by generating it automatically based on requirements. Recently, OpenAI's language model ChatGPT has emerged as a powerful tool for generating human-like responses to a wide range of textual inputs (i.e., prompts), including those related to code generation. However, the effectiveness of ChatGPT for code generation is not well understood, and the generation performance could be heavily influenced by the choice of prompt. To answer these questions, we conducted experiments using the CodeXGlue dataset to evaluate ChatGPT's capabilities for two code generation tasks, including text-to-code and code-to-code generation. We designed prompts by leveraging the chain-of-thought strategy with multi-step optimizations. Our results showed that by carefully designing prompts to guide ChatGPT, the generation performance can be improved substantially. We also analyzed the factors that influenced the prompt design and provided insights that could guide future research.