Investigating the performance of Retrieval-Augmented Generation and fine-tuning for the development of AI-driven knowledge-based systems

TL;DR

This paper compares Retrieval-Augmented Generation (RAG) and fine-tuning (FN) techniques for large language models, demonstrating RAG's superior efficiency and performance in knowledge-based systems, especially in reducing hallucinations.

Contribution

The study provides a comprehensive comparison of RAG and FN for multiple LLMs, highlighting RAG's advantages and proposing a simple architecture that outperforms FN in key metrics.

Findings

RAG outperforms FN by 16% in ROUGE score

RAG achieves 15% higher BLEU scores than FN

RAG reduces hallucinations significantly compared to FN

Abstract

The development of generative large language models (G-LLM) opened up new opportunities for the development of new types of knowledge-based systems similar to ChatGPT, Bing, or Gemini. Fine-tuning (FN) and Retrieval-Augmented Generation (RAG) are the techniques that can be used to implement domain adaptation for the development of G-LLM-based knowledge systems. In our study, using ROUGE, BLEU, METEOR scores, and cosine similarity, we compare and examine the performance of RAG and FN for the GPT-J-6B, OPT-6.7B, LlaMA, LlaMA-2 language models. Based on measurements shown on different datasets, we demonstrate that RAG-based constructions are more efficient than models produced with FN. We point out that connecting RAG and FN is not trivial, because connecting FN models with RAG can cause a decrease in performance. Furthermore, we outline a simple RAG-based architecture which, on average, outperforms the FN models by 16% in terms of the ROGUE score, 15% in the case of the BLEU score, and 53% based on the cosine similarity. This shows the significant advantage of RAG over FN in terms of hallucination, which is not offset by the fact that the average 8% better METEOR score of FN models indicates greater creativity compared to RAG.