LLM-Driven Cost-Effective Requirements Change Impact Analysis

TL;DR

This paper introduces ProReFiCIA, an LLM-based method for automatically identifying impacted requirements due to changes, achieving high recall with minimal review effort and enhanced by domain knowledge integration.

Contribution

It presents a novel LLM-driven approach for impact analysis in requirements management, demonstrating effectiveness and cost-efficiency on industrial datasets.

Findings

Achieves 85.7% recall with minimal review of 3.0% of requirements.

Incorporating domain knowledge increases recall to 95.7%.

Cost remains low, making it practical for real-world use.

Abstract

Requirements are inherently subject to changes throughout the software development lifecycle. Within the limited budget available to requirements engineers, manually identifying the impact of such changes on other requirements is both error-prone and effort-intensive. That might lead to overlooked impacted requirements, which, if not properly managed, can cause serious issues in the downstream tasks. Inspired by the growing potential of large language models (LLMs) across diverse domains, we propose ProReFiCIA, an LLM-driven approach for automatically identifying the impacted requirements when changes occur. We conduct an extensive evaluation of ProReFiCIA using several LLMs and prompts variants tailored to this task. Using the best combination of an LLM and a prompt variant, ProReFiCIA achieves a recall of 85.7% on an unseen industrial dataset, demonstrating its effectiveness in identifying impacted requirements. Further, the cost of applying ProReFiCIA remains small, as the engineer only needs to review the predicted impacted requirements, which represent 3.0% of the entire set of requirements. Lastly, incorporating domain knowledge into the model via RAG increases recall to 95.7% while slightly raising the cost to only 3.6%.