From Questions to Insightful Answers: Building an Informed Chatbot for University Resources

TL;DR

This paper introduces BARKPLUG V.2, an LLM-based chatbot utilizing RAG pipelines to provide accurate, domain-specific campus resource information for university users, enhancing accessibility and user satisfaction.

Contribution

It presents a novel RAG-based chatbot system tailored for university resource inquiries, demonstrating high accuracy and usability in a real-world academic setting.

Findings

Mean RAGAS score of 0.96 indicating high accuracy

High user satisfaction as per SUS surveys

Effective retrieval of campus resource information

Abstract

This paper presents BARKPLUG V.2, a Large Language Model (LLM)-based chatbot system built using Retrieval Augmented Generation (RAG) pipelines to enhance the user experience and access to information within academic settings.The objective of BARKPLUG V.2 is to provide information to users about various campus resources, including academic departments, programs, campus facilities, and student resources at a university setting in an interactive fashion. Our system leverages university data as an external data corpus and ingests it into our RAG pipelines for domain-specific question-answering tasks. We evaluate the effectiveness of our system in generating accurate and pertinent responses for Mississippi State University, as a case study, using quantitative measures, employing frameworks such as Retrieval Augmented Generation Assessment(RAGAS). Furthermore, we evaluate the usability of this system via subjective satisfaction surveys using the System Usability Scale (SUS). Our system demonstrates impressive quantitative performance, with a mean RAGAS score of 0.96, and experience, as validated by usability assessments.