ARAGOG: Advanced RAG Output Grading

TL;DR

This paper evaluates various Retrieval-Augmented Generation methods, highlighting the effectiveness of Hypothetical Document Embedding and Sentence Window Retrieval in improving retrieval precision, and introduces the ARAGOG framework for RAG output grading.

Contribution

It provides an extensive experimental comparison of RAG techniques, identifying the most effective methods and introducing the ARAGOG framework for output grading.

Findings

HyDE and LLM reranking improve retrieval precision

MMR and Cohere rerank show no significant advantage

Sentence Window Retrieval is most effective for retrieval

Abstract

Retrieval-Augmented Generation (RAG) is essential for integrating external knowledge into Large Language Model (LLM) outputs. While the literature on RAG is growing, it primarily focuses on systematic reviews and comparisons of new state-of-the-art (SoTA) techniques against their predecessors, with a gap in extensive experimental comparisons. This study begins to address this gap by assessing various RAG methods' impacts on retrieval precision and answer similarity. We found that Hypothetical Document Embedding (HyDE) and LLM reranking significantly enhance retrieval precision. However, Maximal Marginal Relevance (MMR) and Cohere rerank did not exhibit notable advantages over a baseline Naive RAG system, and Multi-query approaches underperformed. Sentence Window Retrieval emerged as the most effective for retrieval precision, despite its variable performance on answer similarity. The study confirms the potential of the Document Summary Index as a competent retrieval approach. All resources related to this research are publicly accessible for further investigation through our GitHub repository ARAGOG (https://github.com/predlico/ARAGOG). We welcome the community to further this exploratory study in RAG systems.