RAG vs. GraphRAG: A Systematic Evaluation and Key Insights

TL;DR

This paper systematically compares Retrieval-Augmented Generation (RAG) and GraphRAG on text tasks, providing a unified evaluation framework, analyzing their strengths and limitations, and proposing strategies for combining their benefits.

Contribution

It introduces a standardized evaluation protocol for RAG and GraphRAG, conducts a comprehensive benchmark study, and explores methods to integrate both approaches for improved performance.

Findings

RAG excels in certain question answering tasks.

GraphRAG shows advantages in reasoning and grounding.

Combining RAG and GraphRAG yields performance gains.

Abstract

Retrieval-Augmented Generation (RAG) improves large language models (LLMs) by retrieving relevant information from external sources and has been widely adopted for text-based tasks. For structured data, such as knowledge graphs, Graph Retrieval-Augmented Generation (GraphRAG) retrieves and aggregates information along graph structures. More recently, GraphRAG has been extended to general text settings by organizing unstructured text into graph representations, showing promise for reasoning and grounding. Despite these advances, existing GraphRAG systems for text data are often tailored to specific tasks, datasets, and system designs, resulting in heterogeneous evaluation protocols. Consequently, a systematic understanding of the relative strengths, limitations, and trade-offs between RAG and GraphRAG on widely used text benchmarks remains limited. In this paper, we present a comprehensive benchmark study comparing RAG and GraphRAG on established text-based tasks, including question answering and query-based summarization. We introduce a unified evaluation protocol that standardizes data preprocessing, retrieval configurations, and generation settings, enabling fair and reproducible comparisons. Our results highlight the distinct strengths of RAG and GraphRAG across different tasks and evaluation perspectives. Building on these findings, we explore selection and integration strategies that combine the strengths of both paradigms, leading to consistent performance improvements. We further analyze failure modes, efficiency trade-offs, and evaluation biases, and highlight key considerations for designing and evaluating retrieval-augmented generation systems.