Which Prompting Technique Should I Use? An Empirical Investigation of Prompting Techniques for Software Engineering Tasks

TL;DR

This paper systematically evaluates 14 prompt engineering techniques across 10 software engineering tasks using four LLMs, providing insights into their effectiveness, efficiency, and suitability for different types of SE tasks.

Contribution

It offers a comprehensive empirical comparison of prompting techniques for SE tasks, highlighting their relative strengths and guiding practitioners in selecting optimal methods.

Findings

Certain prompting techniques outperform others depending on task complexity.

Prompt linguistic features correlate with effectiveness.

Time and token costs vary significantly across techniques.

Abstract

A growing variety of prompt engineering techniques has been proposed for Large Language Models (LLMs), yet systematic evaluation of each technique on individual software engineering (SE) tasks remains underexplored. In this study, we present a systematic evaluation of 14 established prompt techniques across 10 SE tasks using four LLM models. As identified in the prior literature, the selected prompting techniques span six core dimensions (Zero-Shot, Few-Shot, Thought Generation, Ensembling, Self-Criticism, and Decomposition). They are evaluated on tasks such as code generation, bug fixing, and code-oriented question answering, to name a few. Our results show which prompting techniques are most effective for SE tasks requiring complex logic and intensive reasoning versus those that rely more on contextual understanding and example-driven scenarios. We also analyze correlations between the linguistic characteristics of prompts and the factors that contribute to the effectiveness of prompting techniques in enhancing performance on SE tasks. Additionally, we report the time and token consumption for each prompting technique when applied to a specific task and model, offering guidance for practitioners in selecting the optimal prompting technique for their use cases.