From Bugs to Benefits: Improving User Stories by Leveraging Crowd Knowledge with CrUISE-AC

TL;DR

This paper introduces CrUISE-AC, an automated NLP-based tool that leverages issue tracker data to enhance user stories with additional acceptance criteria, improving requirements quality and reducing defect costs.

Contribution

The paper presents a novel automated method, CrUISE-AC, that uses issue tracker data and NLP techniques to generate valuable acceptance criteria for user stories.

Findings

80-82% of generated criteria are relevant to requirements.

Issue trackers contain valuable domain-specific information.

CrUISE-AC effectively enhances user stories with minimal manual effort.

Abstract

Costs for resolving software defects increase exponentially in late stages. Incomplete or ambiguous requirements are one of the biggest sources for defects, since stakeholders might not be able to communicate their needs or fail to share their domain specific knowledge. Combined with insufficient developer experience, teams are prone to constructing incorrect or incomplete features. To prevent this, requirements engineering has to explore knowledge sources beyond stakeholder interviews. Publicly accessible issue trackers for systems within the same application domain hold essential information on identified weaknesses, edge cases, and potential error sources, all documented by actual users. Our research aims at (1) identifying, and (2) leveraging such issues to improve an agile requirements artifact known as a "user story". We present CrUISE-AC (Crowd and User Informed Suggestion Engine for Acceptance Criteria) as a fully automated method that investigates issues and generates non-trivial additional acceptance criteria for a given user story by employing NLP techniques and an ensemble of LLMs. CrUISE- AC was evaluated by five independent experts in two distinct business domains. Our findings suggest that issue trackers hold valuable information pertinent to requirements engineering. Our evaluation shows that 80-82% of the generated acceptance criteria add relevant requirements to the user stories. Limitations are the dependence on accessible input issues and the fact that we do not check generated criteria for being conflict-free or non-overlapping with criteria from other user stories.