Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A Case Study on Domain-Specific Queries in Private Knowledge-Bases

TL;DR

This paper presents a RAG-based system to enhance the factual accuracy of LLMs for domain-specific queries, addressing hallucinations and leveraging external knowledge bases.

Contribution

It introduces an end-to-end RAG system with dataset processing and evaluation, demonstrating improved accuracy for private, domain-specific, and time-sensitive questions.

Findings

RAG improves LLM factual accuracy in domain-specific tasks.

Fine-tuning with curated datasets has limitations on small or skewed data.

System effectively reduces hallucinations in knowledge-intensive queries.

Abstract

We proposed an end-to-end system design towards utilizing Retrieval Augmented Generation (RAG) to improve the factual accuracy of Large Language Models (LLMs) for domain-specific and time-sensitive queries related to private knowledge-bases. Our system integrates RAG pipeline with upstream datasets processing and downstream performance evaluation. Addressing the challenge of LLM hallucinations, we finetune models with a curated dataset which originates from CMU's extensive resources and annotated with the teacher model. Our experiments demonstrate the system's effectiveness in generating more accurate answers to domain-specific and time-sensitive inquiries. The results also revealed the limitations of fine-tuning LLMs with small-scale and skewed datasets. This research highlights the potential of RAG systems in augmenting LLMs with external datasets for improved performance in knowledge-intensive tasks. Our code and models are available on Github.