KGRAG-Ex: Explainable Retrieval-Augmented Generation with Knowledge Graph-based Perturbations

TL;DR

KGRAG-Ex enhances retrieval-augmented generation by integrating knowledge graphs for improved factual grounding and explainability, using perturbation methods to analyze the influence of graph components on generated responses.

Contribution

This work introduces KGRAG-Ex, a novel system that leverages domain-specific knowledge graphs and perturbation techniques to improve explainability in retrieval-augmented language models.

Findings

Perturbation methods reveal the influence of KG components on answers.

Graph structure impacts the importance of entities in explanations.

Semantic node types affect the interpretability of retrieval paths.

Abstract

Retrieval-Augmented Generation (RAG) enhances language models by grounding responses in external information, yet explainability remains a critical challenge, particularly when retrieval relies on unstructured text. Knowledge graphs (KGs) offer a solution by introducing structured, semantically rich representations of entities and their relationships, enabling transparent retrieval paths and interpretable reasoning. In this work, we present KGRAG-Ex, a RAG system that improves both factual grounding and explainability by leveraging a domain-specific KG constructed via prompt-based information extraction. Given a user query, KGRAG-Ex identifies relevant entities and semantic paths in the graph, which are then transformed into pseudo-paragraphs: natural language representations of graph substructures that guide corpus retrieval. To improve interpretability and support reasoning transparency, we incorporate perturbation-based explanation methods that assess the influence of specific KG-derived components on the generated answers. We conduct a series of experiments to analyze the sensitivity of the system to different perturbation methods, the relationship between graph component importance and their structural positions, the influence of semantic node types, and how graph metrics correspond to the influence of components within the explanations process.