A Case Study Investigating the Role of Generative AI in Quality Evaluations of Epics in Agile Software Development

TL;DR

This case study explores how large language models can evaluate the quality of agile epics, revealing potential benefits, challenges, and user perceptions in a real-world industry setting.

Contribution

It provides empirical insights into integrating generative AI into agile epic quality assessments, highlighting practical applications and barriers.

Findings

LLMs can effectively evaluate epic quality with high user satisfaction.

Users perceive AI evaluations as valuable for improving epics.

Challenges include adoption barriers and limitations of current AI models.

Abstract

The broad availability of generative AI offers new opportunities to support various work domains, including agile software development. Agile epics are a key artifact for product managers to communicate requirements to stakeholders. However, in practice, they are often poorly defined, leading to churn, delivery delays, and cost overruns. In this industry case study, we investigate opportunities for large language models (LLMs) to evaluate agile epic quality in a global company. Results from a user study with 17 product managers indicate how LLM evaluations could be integrated into their work practices, including perceived values and usage in improving their epics. High levels of satisfaction indicate that agile epics are a new, viable application of AI evaluations. However, our findings also outline challenges, limitations, and adoption barriers that can inform both practitioners and researchers on the integration of such evaluations into future agile work practices.