Towards Requirements Engineering for RAG Systems

TL;DR

This paper investigates the requirements engineering process for Retrieval Augmented Generation (RAG) systems in a maritime setting, highlighting the iterative collaboration needed between data scientists and users to define correctness and system requirements.

Contribution

It introduces an empirical process model for eliciting retrieval requirements in RAG systems within complex, domain-specific applications.

Findings

Data scientists face a tension between user expectations and output correctness.

Iterative experimentation with users helps define context-specific retrieval requirements.

The process model guides practical requirements elicitation for RAG systems.

Abstract

This short paper explores how a maritime company develops and integrates large-language models (LLM). Specifically by looking at the requirements engineering for Retrieval Augmented Generation (RAG) systems in expert settings. Through a case study at a maritime service provider, we demonstrate how data scientists face a fundamental tension between user expectations of AI perfection and the correctness of the generated outputs. Our findings reveal that data scientists must identify context-specific "retrieval requirements" through iterative experimentation together with users because they are the ones who can determine correctness. We present an empirical process model describing how data scientists practically elicited these "retrieval requirements" and managed system limitations. This work advances software engineering knowledge by providing insights into the specialized requirements engineering processes for implementing RAG systems in complex domain-specific applications.