More Than a Score: Probing the Impact of Prompt Specificity on LLM Code Generation

TL;DR

This paper investigates how the level of prompt detail affects large language models' code generation performance, revealing that increased prompt specificity can significantly improve results depending on the task.

Contribution

Introduces PartialOrderEval, a benchmark to systematically study the impact of prompt detail on LLM code generation across multiple datasets.

Findings

Prompt sensitivity varies across tasks and models.

Explicit I/O and stepwise prompts enhance performance.

Prompt detail can significantly improve pass@1 scores.

Abstract

State-of-the-art Large Language Models (LLMs) achieve high pass@1 on general benchmarks like HumanEval but underperform on specialized suites such as ParEval. Is this due to LLMs missing domain knowledge or insufficient prompt detail is given? To answer this, we introduce PartialOrderEval, which augments any code generation benchmark with a partial order of prompts from minimal to maximally detailed. Applying it to HumanEval and both serial and OpenMP subsets of ParEval, we measure how pass@1 scales with prompt specificity. Our experiments with Llama-3.x and Qwen2.5-Coder demonstrate varying degrees of prompt sensitivity across different tasks, and a qualitative analysis highlights explicit I/O specifications, edge-case handling, and stepwise breakdowns as the key drivers of prompt detail improvement.