Towards Robust Retrieval-Augmented Generation Based on Knowledge Graph: A Comparative Analysis

TL;DR

This paper evaluates the robustness of retrieval-augmented generation systems using a new benchmark, comparing a baseline with a knowledge graph-based system and proposing improvements for real-world reliability.

Contribution

It introduces the RGB benchmark for robustness evaluation and compares RAG systems, including a novel GraphRAG approach with enhancements.

Findings

GraphRAG outperforms the baseline in robustness tests.

Customizations improve retrieval accuracy and response reliability.

Insights guide the design of more dependable RAG systems.

Abstract

Retrieval-Augmented Generation (RAG) was introduced to enhance the capabilities of Large Language Models (LLMs) beyond their encoded prior knowledge. This is achieved by providing LLMs with an external source of knowledge, which helps reduce factual hallucinations and enables access to new information not available during pretraining. However, inconsistent retrieved information can negatively affect LLM responses. The Retrieval-Augmented Generation Benchmark (RGB) was introduced to evaluate the robustness of RAG systems under such conditions. In this work, we use the RGB corpus to evaluate LLMs in four scenarios: noise robustness, information integration, negative rejection, and counterfactual robustness. We perform a comparative analysis between the RGB RAG baseline and GraphRAG, a knowledge graph based retrieval system. We test three GraphRAG customizations to improve robustness. Results show improvements over the RGB baseline and provide insights for designing more reliable RAG systems for real world scenarios.