A RAG-Based Institutional Assistant
Gustavo Kuratomi, Paulo Pirozelli, Fabio G. Cozman, Sarajane M. Peres

TL;DR
This paper presents a retrieval-augmented generation system tailored for a university assistant, demonstrating significant performance improvements when relevant documents are provided to large language models, highlighting the importance of effective retrieval.
Contribution
The paper designs and evaluates a RAG-based virtual assistant for a university, analyzing the impact of retrieval quality on LLM performance in knowledge-intensive tasks.
Findings
Optimal retriever achieves 30% Top-5 accuracy
Providing correct documents boosts accuracy to 54.02%
Without context, accuracy drops to 13.68%
Abstract
Although large language models (LLMs) demonstrate strong text generation capabilities, they struggle in scenarios requiring access to structured knowledge bases or specific documents, limiting their effectiveness in knowledge-intensive tasks. To address this limitation, retrieval-augmented generation (RAG) models have been developed, enabling generative models to incorporate relevant document fragments into their inputs. In this paper, we design and evaluate a RAG-based virtual assistant specifically tailored for the University of S\~ao Paulo. Our system architecture comprises two key modules: a retriever and a generative model. We experiment with different types of models for both components, adjusting hyperparameters such as chunk size and the number of retrieved documents. Our optimal retriever model achieves a Top-5 accuracy of 30%, while our most effective generative model scores…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsBusiness Process Modeling and Analysis · Advanced Software Engineering Methodologies
