XGraphRAG: Interactive Visual Analysis for Graph-based Retrieval-Augmented Generation

TL;DR

XGraphRAG is a visual analysis tool designed to help developers understand and improve graph-based retrieval-augmented generation systems by providing interactive visualizations of the complex pipeline and recall processes.

Contribution

The paper introduces XGraphRAG, a novel visual analysis framework and prototype system that enhances interpretability and debugging of GraphRAG models.

Findings

Effective identification of recall issues in GraphRAG

Improved understanding of GraphRAG pipeline processes

Enhanced debugging and model improvement capabilities

Abstract

Graph-based Retrieval-Augmented Generation (RAG) has shown great capability in enhancing Large Language Model (LLM)'s answer with an external knowledge base. Compared to traditional RAG, it introduces a graph as an intermediate representation to capture better structured relational knowledge in the corpus, elevating the precision and comprehensiveness of generation results. However, developers usually face challenges in analyzing the effectiveness of GraphRAG on their dataset due to GraphRAG's complex information processing pipeline and the overwhelming amount of LLM invocations involved during graph construction and query, which limits GraphRAG interpretability and accessibility. This research proposes a visual analysis framework that helps RAG developers identify critical recalls of GraphRAG and trace these recalls through the GraphRAG pipeline. Based on this framework, we develop XGraphRAG, a prototype system incorporating a set of interactive visualizations to facilitate users' analysis process, boosting failure cases collection and improvement opportunities identification. Our evaluation demonstrates the effectiveness and usability of our approach. Our work is open-sourced and available at https://github.com/Gk0Wk/XGraphRAG.