Generative AI-Based Virtual Assistant using Retrieval-Augmented Generation: An evaluation study for bachelor projects

TL;DR

This paper presents a retrieval-augmented generative AI virtual assistant tailored for university students, improving accuracy and reliability in specialized educational contexts through domain-specific knowledge integration.

Contribution

It introduces a novel retrieval-augmented generation system for educational virtual assistants, addressing hallucinations and information gaps in domain-specific applications.

Findings

The virtual assistant effectively supports students with project regulations.

Enhanced response accuracy and context relevance demonstrated through evaluation.

Addresses challenges of LLM hallucinations in specialized domains.

Abstract

Large Language Models have been increasingly employed in the creation of Virtual Assistants due to their ability to generate human-like text and handle complex inquiries. While these models hold great promise, challenges such as hallucinations, missing information, and the difficulty of providing accurate and context-specific responses persist, particularly when applied to highly specialized content domains. In this paper, we focus on addressing these challenges by developing a virtual assistant designed to support students at Maastricht University in navigating project-specific regulations. We propose a virtual assistant based on a Retrieval-Augmented Generation system that enhances the accuracy and reliability of responses by integrating up-to-date, domain-specific knowledge. Through a robust evaluation framework and real-life testing, we demonstrate that our virtual assistant can effectively meet the needs of students while addressing the inherent challenges of applying Large Language Models to a specialized educational context. This work contributes to the ongoing discourse on improving LLM-based systems for specific applications and highlights areas for further research.