KGEval: Estimating Accuracy of Automatically Constructed Knowledge Graphs

TL;DR

KGEval is a novel method that estimates the accuracy of large, automatically constructed knowledge graphs by leveraging coupling constraints and crowdsourcing, achieving higher accuracy with fewer human evaluations.

Contribution

This paper introduces KGEval, a new approach that efficiently estimates KG accuracy using coupling constraints and crowdsourcing, with theoretical guarantees and superior performance.

Findings

KGEval outperforms baseline methods in accuracy estimation.

KGEval requires fewer human evaluations than existing approaches.

The objective function in KGEval is submodular and NP-hard, with approximation guarantees.

Abstract

Automatic construction of large knowledge graphs (KG) by mining web-scale text datasets has received considerable attention recently. Estimating accuracy of such automatically constructed KGs is a challenging problem due to their size and diversity. This important problem has largely been ignored in prior research we fill this gap and propose KGEval. KGEval binds facts of a KG using coupling constraints and crowdsources the facts that infer correctness of large parts of the KG. We demonstrate that the objective optimized by KGEval is submodular and NP-hard, allowing guarantees for our approximation algorithm. Through extensive experiments on real-world datasets, we demonstrate that KGEval is able to estimate KG accuracy more accurately compared to other competitive baselines, while requiring significantly lesser number of human evaluations.