From What to How: Bridging User Requirements with Software Development Using Large Language Models

TL;DR

This paper introduces DesBench, a benchmark for evaluating large language models on software design tasks, revealing current limitations in handling design intricacies despite progress in code generation.

Contribution

The paper presents DesBench, a novel design-aware benchmark for assessing LLMs on software design, object-oriented modeling, and acceptance test generation, filling a gap in existing evaluation methods.

Findings

LLMs struggle with design-aware code generation from high-level requirements.

LLMs can identify objects and classes but have difficulty defining operations and relationships.

Generated acceptance tests achieve human-like code coverage quality.

Abstract

Recently, large language models (LLMs) are extensively utilized to enhance development efficiency, leading to numerous benchmarks for evaluating their performance. However, these benchmarks predominantly focus on implementation, overlooking the equally critical aspect of software design. This gap raises two pivotal questions: (1) Can LLMs handle software design? (2) Can LLMs write code following the specific designs? To investigate these questions, this paper proposes DesBench, a design-aware benchmark for evaluating LLMs on three software design-related tasks: design-aware code generation, object-oriented modeling, and the design of acceptance test cases. DesBench comprises 30 manually crafted Java projects that include requirement documents, design models, implementations, and acceptance tests, amounting to a total of 30 design models, 194 Java classes, and 737 test cases. We evaluated seven state-of-the-art LLMs, including three DeepSeek R1, two Qwen2.5, and two GPT models, using DesBench. The results reveal that LLMs remain significantly challenged by the intricacies of software design: (1) For code generation, LLMs struggle to produce correct implementations when provided with only high-level or no designs. (2) In object-oriented modeling, while LLMs can accurately identify objects and classes, they face challenges in defining operations and inter-class relationships. (3) Acceptance test cases generated by LLMs from functional requirements achieve code coverage quality comparable to those written by humans. Our research highlights the current limitations of LLMs in managing software design and calls for further investigation into new design methodologies and languages suitable for LLM-based development.