Rule2Text: Natural Language Explanation of Logical Rules in Knowledge Graphs

TL;DR

This paper investigates using large language models to generate natural language explanations for logical rules in knowledge graphs, aiming to improve interpretability and understanding of complex data patterns.

Contribution

It introduces a method for automatically translating logical rules from knowledge graphs into natural language explanations using LLMs, evaluated through comprehensive human and automatic assessments.

Findings

Promising accuracy in correctness and clarity of explanations

Effective prompting strategies including chain-of-thought reasoning

Identification of challenges like hallucination in generated explanations

Abstract

Knowledge graphs (KGs) often contain sufficient information to support the inference of new facts. Identifying logical rules not only improves the completeness of a knowledge graph but also enables the detection of potential errors, reveals subtle data patterns, and enhances the overall capacity for reasoning and interpretation. However, the complexity of such rules, combined with the unique labeling conventions of each KG, can make them difficult for humans to understand. In this paper, we explore the potential of large language models to generate natural language explanations for logical rules. Specifically, we extract logical rules using the AMIE 3.5.1 rule discovery algorithm from the benchmark dataset FB15k-237 and two large-scale datasets, FB-CVT-REV and FB+CVT-REV. We examine various prompting strategies, including zero- and few-shot prompting, including variable entity types, and chain-of-thought reasoning. We conduct a comprehensive human evaluation of the generated explanations based on correctness, clarity, and hallucination, and also assess the use of large language models as automatic judges. Our results demonstrate promising performance in terms of explanation correctness and clarity, although several challenges remain for future research. All scripts and data used in this study are publicly available at https://github.com/idirlab/KGRule2NL}{https://github.com/idirlab/KGRule2NL.