Multi-Modal Requirements Data-based Acceptance Criteria Generation using LLMs

TL;DR

This paper introduces RAGcceptance M2RE, a novel multi-modal retrieval-augmented generation approach that leverages textual and visual UI data to automatically generate accurate acceptance criteria, improving software validation efficiency.

Contribution

It presents a new multi-modal RAG-based method for generating acceptance criteria from combined textual and visual requirements data, demonstrating practical benefits in an industrial case study.

Findings

Enhanced relevance and correctness of generated ACs

Reduced manual effort in criteria creation

Practitioner validation confirms practical utility

Abstract

Acceptance criteria (ACs) play a critical role in software development by clearly defining the conditions under which a software feature satisfies stakeholder expectations. However, manually creating accurate, comprehensive, and unambiguous acceptance criteria is challenging, particularly in user interface-intensive applications, due to the reliance on domain-specific knowledge and visual context that is not always captured by textual requirements alone. To address these challenges, we propose RAGcceptance M2RE, a novel approach that leverages Retrieval-Augmented Generation (RAG) to generate acceptance criteria from multi-modal requirements data, including both textual documentation and visual UI information. We systematically evaluated our approach in an industrial case study involving an education-focused software system used by approximately 100,000 users. The results indicate that integrating multi-modal information significantly enhances the relevance, correctness, and comprehensibility of the generated ACs. Moreover, practitioner evaluations confirm that our approach effectively reduces manual effort, captures nuanced stakeholder intent, and provides valuable criteria that domain experts may overlook, demonstrating practical utility and significant potential for industry adoption. This research underscores the potential of multi-modal RAG techniques in streamlining software validation processes and improving development efficiency. We also make our implementation and a dataset available.