Efficient Knowledge Graph Accuracy Evaluation

TL;DR

This paper introduces an efficient sampling framework for large-scale knowledge graph accuracy evaluation, significantly reducing human annotation costs while maintaining statistical reliability.

Contribution

It proposes novel sampling strategies, including cluster, stratified, and weighted sampling, for cost-effective and incremental accuracy evaluation of evolving knowledge graphs.

Findings

Up to 60% cost reduction in static KG evaluation

Up to 80% cost reduction in evolving KG evaluation

Maintains evaluation quality with reduced human effort

Abstract

Estimation of the accuracy of a large-scale knowledge graph (KG) often requires humans to annotate samples from the graph. How to obtain statistically meaningful estimates for accuracy evaluation while keeping human annotation costs low is a problem critical to the development cycle of a KG and its practical applications. Surprisingly, this challenging problem has largely been ignored in prior research. To address the problem, this paper proposes an efficient sampling and evaluation framework, which aims to provide quality accuracy evaluation with strong statistical guarantee while minimizing human efforts. Motivated by the properties of the annotation cost function observed in practice, we propose the use of cluster sampling to reduce the overall cost. We further apply weighted and two-stage sampling as well as stratification for better sampling designs. We also extend our framework to enable efficient incremental evaluation on evolving KG, introducing two solutions based on stratified sampling and a weighted variant of reservoir sampling. Extensive experiments on real-world datasets demonstrate the effectiveness and efficiency of our proposed solution. Compared to baseline approaches, our best solutions can provide up to 60% cost reduction on static KG evaluation and up to 80% cost reduction on evolving KG evaluation, without loss of evaluation quality.