DeepQuali: Initial results of a study on the use of large language models for assessing the quality of user stories

TL;DR

DeepQuali leverages GPT-4o to assess and enhance user story quality in agile development, showing promising agreement with experts and highlighting the potential of LLMs in requirements validation.

Contribution

This study introduces DeepQuali, an LLM-based approach for requirements quality assessment, integrating quality models and explanations to support software engineers.

Findings

Experts largely agreed with LLM assessments

The approach was well accepted with explanatory feedback

Potential for LLMs to support requirements validation

Abstract

Generative artificial intelligence (GAI), specifically large language models (LLMs), are increasingly used in software engineering, mainly for coding tasks. However, requirements engineering - particularly requirements validation - has seen limited application of GAI. The current focus of using GAI for requirements is on eliciting, transforming, and classifying requirements, not on quality assessment. We propose and evaluate the LLM-based (GPT-4o) approach "DeepQuali", for assessing and improving requirements quality in agile software development. We applied it to projects in two small companies, where we compared LLM-based quality assessments with expert judgments. Experts also participated in walkthroughs of the solution, provided feedback, and rated their acceptance of the approach. Experts largely agreed with the LLM's quality assessments, especially regarding overall ratings and explanations. However, they did not always agree with the other experts on detailed ratings, suggesting that expertise and experience may influence judgments. Experts recognized the usefulness of the approach but criticized the lack of integration into their workflow. LLMs show potential in supporting software engineers with the quality assessment and improvement of requirements. The explicit use of quality models and explanatory feedback increases acceptance.