Generating Test Scenarios from NL Requirements using Retrieval-Augmented LLMs: An Industrial Study

TL;DR

This paper introduces RAGTAG, an automated method using Retrieval-Augmented Generation with LLMs to generate test scenarios from bilingual requirements, demonstrating promising results in industrial settings.

Contribution

The paper presents RAGTAG, a novel approach combining retrieval-augmented generation with LLMs for automated test scenario generation from bilingual requirements.

Findings

RAGTAG produces scenarios aligned with requirements.

Generated scenarios are understandable and feasible.

Correctness is satisfactory but domain nuances remain challenging.

Abstract

Test scenarios are specific instances of test cases that describe actions to validate a particular software functionality. By outlining the conditions under which the software operates and the expected outcomes, test scenarios ensure that the software functionality is tested in an integrated manner. Test scenarios are crucial for systematically testing an application under various conditions, including edge cases, to identify potential issues and guarantee overall performance and reliability. Specifying test scenarios is tedious and requires a deep understanding of software functionality and the underlying domain. It further demands substantial effort and investment from already time- and budget-constrained requirements engineers and testing teams. This paper presents an automated approach (RAGTAG) for test scenario generation using Retrieval-Augmented Generation (RAG) with Large Language Models (LLMs). RAG allows the integration of specific domain knowledge with LLMs' generation capabilities. We evaluate RAGTAG on two industrial projects from Austrian Post with bilingual requirements in German and English. Our results from an interview survey conducted with four experts on five dimensions -- relevance, coverage, correctness, coherence and feasibility, affirm the potential of RAGTAG in automating test scenario generation. Specifically, our results indicate that, despite the difficult task of analyzing bilingual requirements, RAGTAG is able to produce scenarios that are well-aligned with the underlying requirements and provide coverage of different aspects of the intended functionality. The generated scenarios are easily understandable to experts and feasible for testing in the project environment. The overall correctness is deemed satisfactory; however, gaps in capturing exact action sequences and domain nuances remain, underscoring the need for domain expertise when applying LLMs.