A RAG-Based Institutional Assistant

TL;DR

This paper presents a retrieval-augmented generation system tailored for a university assistant, demonstrating significant performance improvements when relevant documents are provided to large language models, highlighting the importance of effective retrieval.

Contribution

The paper designs and evaluates a RAG-based virtual assistant for a university, analyzing the impact of retrieval quality on LLM performance in knowledge-intensive tasks.

Findings

Optimal retriever achieves 30% Top-5 accuracy

Providing correct documents boosts accuracy to 54.02%

Without context, accuracy drops to 13.68%

Abstract

Although large language models (LLMs) demonstrate strong text generation capabilities, they struggle in scenarios requiring access to structured knowledge bases or specific documents, limiting their effectiveness in knowledge-intensive tasks. To address this limitation, retrieval-augmented generation (RAG) models have been developed, enabling generative models to incorporate relevant document fragments into their inputs. In this paper, we design and evaluate a RAG-based virtual assistant specifically tailored for the University of S\~ao Paulo. Our system architecture comprises two key modules: a retriever and a generative model. We experiment with different types of models for both components, adjusting hyperparameters such as chunk size and the number of retrieved documents. Our optimal retriever model achieves a Top-5 accuracy of 30%, while our most effective generative model scores 22.04\% against ground truth answers. Notably, when the correct document chunks are supplied to the LLMs, accuracy significantly improves to 54.02%, an increase of over 30 percentage points. Conversely, without contextual input, performance declines to 13.68%. These findings highlight the critical role of database access in enhancing LLM performance. They also reveal the limitations of current semantic search methods in accurately identifying relevant documents and underscore the ongoing challenges LLMs face in generating precise responses.