Assessing the Impact of Requirement Ambiguity on LLM-based Function-Level Code Generation

TL;DR

This paper introduces Orchid, a benchmark for evaluating how requirement ambiguity affects LLM-based code generation, revealing that ambiguity significantly hampers performance and highlights the need for ambiguity-aware models.

Contribution

The paper presents Orchid, the first benchmark specifically designed to assess LLM performance on ambiguous requirements, and provides empirical insights into the impact of ambiguity on code generation.

Findings

Ambiguity degrades LLM code generation performance.

Advanced models are more affected by ambiguity.

LLMs often produce divergent implementations for the same ambiguous requirement.

Abstract

Software requirement ambiguity is ubiquitous in real-world development, stemming from the inherent imprecision of natural language and the varying interpretations of stakeholders. While Large Language Models (LLMs) have demonstrated impressive capabilities in generating code from precise specifications, such ambiguity poses a significant obstacle to reliable automated code generation. Existing benchmarks typically assume clear and unambiguous requirements, leaving an empirical gap in understanding how LLMs behave when faced with the inherent uncertainty of real-world software requirements. In this paper, we introduce Orchid, the first code generation benchmark specifically designed with ambiguous requirements. It comprises 1,304 function-level tasks covering four distinct types of ambiguity: lexical, syntactic, semantic, and vagueness. Leveraging this dataset, we conduct the first systematic empirical study to evaluate the impact of requirement ambiguity on LLM-based code generation. Our results demonstrate that ambiguity consistently degrades the performance of all evaluated LLMs, with the most pronounced negative effects observed in highly advanced models. Furthermore, we observe that LLMs frequently produce functionally divergent implementations for the same ambiguous requirement and lack the capability to identify or resolve such ambiguity autonomously. These findings reveal a significant performance gap between clear and ambiguous requirements, underscoring the urgent need for ambiguity-aware techniques in the next generation of automated software engineering tools. The Orchid benchmark is publicly available at https://huggingface.co/datasets/SII-YDD/Orchid.