Systematic Evaluation of Knowledge Graph Repair with Large Language Models

TL;DR

This paper introduces a systematic evaluation framework for knowledge graph repair systems using large language models, emphasizing the importance of prompt design and violation generation for effective repairs.

Contribution

It presents a novel violation-inducing operations mechanism and a comprehensive evaluation method for assessing knowledge graph repair quality.

Findings

Concise prompts improve repair performance.

Large language models can effectively repair knowledge graphs.

Systematic violation generation enables rigorous evaluation.

Abstract

We present a systematic approach for evaluating the quality of knowledge graph repairs with respect to constraint violations defined in shapes constraint language (SHACL). Current evaluation methods rely on \emph{ad hoc} datasets, which limits the rigorous analysis of repair systems in more general settings. Our method addresses this gap by systematically generating violations using a novel mechanism, termed violation-inducing operations (VIOs). We use the proposed evaluation framework to assess a range of repair systems which we build using large language models. We analyze the performance of these systems across different prompting strategies. Results indicate that concise prompts containing both the relevant violated SHACL constraints and key contextual information from the knowledge graph yield the best performance.