RAGVA: Engineering Retrieval Augmented Generation-based Virtual Assistants in Practice

TL;DR

This paper presents a practical guide and study on engineering Retrieval-Augmented Generation (RAG) virtual assistants, highlighting challenges and future directions based on Transurban's experience to advance AI software engineering.

Contribution

It offers a comprehensive engineering guide for RAG virtual assistants and identifies key challenges and future research directions from real-world deployment at Transurban.

Findings

Identified eight engineering challenges in RAGVA development.

Proposed eight future research directions for RAG-based applications.

Provided a step-by-step guide for building conversational RAG applications.

Abstract

Retrieval-augmented generation (RAG)-based applications are gaining prominence due to their ability to leverage large language models (LLMs). These systems excel at combining retrieval mechanisms with generative capabilities, resulting in more accurate, contextually relevant responses that enhance user experience. In particular, Transurban, a road operation company, is replacing its rule-based virtual assistant (VA) with a RAG-based VA (RAGVA) to offer more flexible customer interactions and support a wider range of scenarios. In this paper, drawing from the experience at Transurban, we present a comprehensive step-by-step guide for building a conversational application and how to engineer a RAGVA. These guides aim to serve as references for future researchers and practitioners. While the engineering processes for traditional software applications are well-established, the development and evaluation of RAG-based applications are still in their early stages, with numerous emerging challenges remaining uncharted. To address this gap, we conduct a focus group study with Transurban practitioners regarding developing and evaluating their RAGVA. We identified eight challenges encountered by the engineering team and proposed eight future directions that should be explored to advance the development of RAG-based applications. This study contributes to the foundational understanding of a RAG-based conversational application and the emerging AI software engineering challenges it presents.