Analysis of LLMs vs Human Experts in Requirements Engineering

TL;DR

This study compares Large Language Models to human experts in requirements elicitation, revealing LLMs produce more aligned and complete requirements faster and at a fraction of the cost, indicating their growing role in requirements engineering.

Contribution

It provides empirical evidence on LLMs' effectiveness in requirements elicitation, highlighting their potential to enhance efficiency and quality in requirements engineering.

Findings

LLMs generated requirements more aligned (+1.12) with stakeholder needs.

LLMs produced more complete requirements (+10.2%).

LLMs operated at 720x the speed and at 0.06% of human cost.

Abstract

The majority of research around Large Language Models (LLM) application to software development has been on the subject of code generation. There is little literature on LLMs' impact on requirements engineering (RE), which deals with the process of developing and verifying the system requirements. Within RE, there is a subdiscipline of requirements elicitation, which is the practice of discovering and documenting requirements for a system from users, customers, and other stakeholders. In this analysis, we compare LLM's ability to elicit requirements of a software system, as compared to that of a human expert in a time-boxed and prompt-boxed study. We found LLM-generated requirements were evaluated as more aligned (+1.12) than human-generated requirements with a trend of being more complete (+10.2%). Conversely, we found users tended to believe that solutions they perceived as more aligned had been generated by human experts. Furthermore, while LLM-generated documents scored higher and performed at 720x the speed, their cost was, on average, only 0.06% that of a human expert. Overall, these findings indicate that LLMs will play an increasingly important role in requirements engineering by improving requirements definitions, enabling more efficient resource allocation, and reducing overall project timelines.