Knowledge Graph Quality Evaluation under Incomplete Information

TL;DR

This paper introduces QEII, a novel framework for evaluating knowledge graph quality under incomplete information by transforming the task into an adversarial Q&A game, ensuring data privacy and assessing ability level quality.

Contribution

The paper proposes a new quality evaluation framework that operates without exposing raw data and focuses on ability level assessment, addressing limitations of existing methods.

Findings

QEII effectively evaluates KG quality without raw data exposure

Experimental results show QEII outperforms baselines in ability level assessment

Framework applicable to multiple KG pairs under incomplete information

Abstract

Knowledge graphs (KGs) have attracted more and more attentions because of their fundamental roles in many tasks. Quality evaluation for KGs is thus crucial and indispensable. Existing methods in this field evaluate KGs by either proposing new quality metrics from different dimensions or measuring performances at KG construction stages. However, there are two major issues with those methods. First, they highly rely on raw data in KGs, which makes KGs' internal information exposed during quality evaluation. Second, they consider more about the quality at data level instead of ability level, where the latter one is more important for downstream applications. To address these issues, we propose a knowledge graph quality evaluation framework under incomplete information (QEII). The quality evaluation task is transformed into an adversarial Q&A game between two KGs. Winner of the game is thus considered to have better qualities. During the evaluation process, no raw data is exposed, which ensures information protection. Experimental results on four pairs of KGs demonstrate that, compared with baselines, the QEII implements a reasonable quality evaluation at ability level under incomplete information.