A Pilot Empirical Study on When and How to Use Knowledge Graphs as Retrieval Augmented Generation

TL;DR

This paper systematically evaluates when and how to effectively integrate Knowledge Graphs into Retrieval Augmented Generation, analyzing various configurations across multiple datasets to guide practical application.

Contribution

It provides the first comprehensive empirical analysis of KG-RAG methods, configurations, and scenarios, establishing foundational insights for future research and application.

Findings

Proper application conditions are crucial for KG-RAG effectiveness.

Optimal configurations significantly impact performance across scenarios.

Combining Metacognition with KG-RAG shows promising results.

Abstract

The integration of Knowledge Graphs (KGs) into the Retrieval Augmented Generation (RAG) framework has attracted significant interest, with early studies showing promise in mitigating hallucinations and improving model accuracy. However, a systematic understanding and comparative analysis of the rapidly emerging KG-RAG methods are still lacking. This paper seeks to lay the foundation for systematically answering the question of when and how to use KG-RAG by analyzing their performance in various application scenarios associated with different technical configurations. After outlining the mind map using KG-RAG framework and summarizing its popular pipeline, we conduct a pilot empirical study of KG-RAG works to reimplement and evaluate 6 KG-RAG methods across 9 datasets in diverse domains and scenarios, analyzing the impact of 9 KG-RAG configurations in combination with 17 LLMs, and combining Metacognition with KG-RAG as a pilot attempt. Our results underscore the critical role of appropriate application conditions and optimal configurations of KG-RAG components.