How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAG

TL;DR

This paper introduces an unbiased evaluation framework for GraphRAG that addresses flaws in current assessment methods, revealing more moderate performance gains of existing models and emphasizing the need for rigorous evaluation in the field.

Contribution

The paper proposes a novel unbiased evaluation framework for GraphRAG that improves question relevance and answer assessment accuracy, leading to more reliable performance evaluations.

Findings

Performance gains of GraphRAG are more moderate than previously reported.

The new evaluation framework reduces biases in answer quality assessment.

Current evaluation methods may overestimate GraphRAG capabilities.

Abstract

By retrieving contexts from knowledge graphs, graph-based retrieval-augmented generation (GraphRAG) enhances large language models (LLMs) to generate quality answers for user questions. Many GraphRAG methods have been proposed and reported inspiring performance in answer quality. However, we observe that the current answer evaluation framework for GraphRAG has two critical flaws, i.e., unrelated questions and evaluation biases, which may lead to biased or even wrong conclusions on performance. To tackle the two flaws, we propose an unbiased evaluation framework that uses graph-text-grounded question generation to produce questions that are more related to the underlying dataset and an unbiased evaluation procedure to eliminate the biases in LLM-based answer assessment. We apply our unbiased framework to evaluate 3 representative GraphRAG methods and find that their performance gains are much more moderate than reported previously. Although our evaluation framework may still have flaws, it calls for scientific evaluations to lay solid foundations for GraphRAG research.