An Analysis of Automated Use Case Component Extraction from Scenarios using ChatGPT

TL;DR

This paper presents a method leveraging large language models to automatically extract use case components from user scenarios, aiding requirements analysis in mobile app development.

Contribution

It introduces a prompt-based approach that incorporates domain knowledge to improve the accuracy of use case component extraction from scenarios.

Findings

Refined prompts with domain knowledge enhance extraction precision.

LLMs need domain-specific context for better use case component identification.

The method achieves improved recall and precision on a labeled dataset.

Abstract

Mobile applications (apps) are often developed by only a small number of developers with limited resources, especially in the early years of the app's development. In this setting, many requirements acquisition activities, such as interviews, are challenging or lower priority than development and release activities. Moreover, in this early period, requirements are frequently changing as mobile apps evolve to compete in the marketplace. As app development companies move to standardize their development processes, however, they will shift to documenting and analyzing requirements. One low-cost source of requirements post-deployment are user-authored scenarios describing how they interact with an app. We propose a method for extracting use case components from user-authored scenarios using large language models (LLMs). The method consists of a series of prompts that were developed to improve precision and recall on a ground truth dataset of 50 scenarios independently labeled with UC components. Our results reveal that LLMs require additional domain knowledge to extract UC components, and that refining prompts to include this knowledge improves the quality of the extracted UC components.