Do Advanced Language Models Eliminate the Need for Prompt Engineering in Software Engineering?

TL;DR

This study empirically evaluates whether advanced LLMs eliminate the need for prompt engineering in software engineering tasks, finding that simpler prompts often suffice and reasoning models excel mainly in complex tasks.

Contribution

It provides a comprehensive empirical analysis of prompt engineering effectiveness across advanced LLMs in software engineering, highlighting when complex prompts are beneficial or unnecessary.

Findings

Prompt engineering benefits diminish with advanced models.

Reasoning models excel in complex tasks but add little in simple ones.

Simple zero-shot prompts can outperform complex engineered prompts.

Abstract

Large Language Models (LLMs) have significantly advanced software engineering (SE) tasks, with prompt engineering techniques enhancing their performance in code-related areas. However, the rapid development of foundational LLMs such as the non-reasoning model GPT-4o and the reasoning model o1 raises questions about the continued effectiveness of these prompt engineering techniques. This paper presents an extensive empirical study that reevaluates various prompt engineering techniques within the context of these advanced LLMs. Focusing on three representative SE tasks, i.e., code generation, code translation, and code summarization, we assess whether prompt engineering techniques still yield improvements with advanced models, the actual effectiveness of reasoning models compared to non-reasoning models, and whether the benefits of using these advanced models justify their increased costs. Our findings reveal that prompt engineering techniques developed for earlier LLMs may provide diminished benefits or even hinder performance when applied to advanced models. In reasoning LLMs, the ability of sophisticated built-in reasoning reduces the impact of complex prompts, sometimes making simple zero-shot prompting more effective. Furthermore, while reasoning models outperform non-reasoning models in tasks requiring complex reasoning, they offer minimal advantages in tasks that do not need reasoning and may incur unnecessary costs. Based on our study, we provide practical guidance for practitioners on selecting appropriate prompt engineering techniques and foundational LLMs, considering factors such as task requirements, operational costs, and environmental impact. Our work contributes to a deeper understanding of effectively harnessing advanced LLMs in SE tasks, informing future research and application development.