A Protocol for KG Construction Tasks Involving Users

TL;DR

This paper introduces a standardized user protocol for knowledge graph construction tasks to enable consistent comparison of languages, techniques, and tools, addressing current heterogeneity in evaluation methods.

Contribution

It proposes a systematic user protocol for KGC that facilitates comparison across different languages and tools, filling a gap in standardized evaluation methods.

Findings

The protocol enables comparison of RML core functionalities.

It allows for evaluation of extensions to RML.

It promotes systematic analysis and reporting in KGC user studies.

Abstract

Knowledge graph construction (KGC) from (semi-)structured data is challenging, and facilitating user involvement is an issue frequently brought up within this community. We cannot deny the progress we have made with respect to (declarative) knowledge graph construction languages and tools to help build such mappings. However, it is surprising that no two studies report on similar protocols. This heterogeneity does not allow for comparing KGC languages, techniques, and tools. This paper first analyses studies involving users to identify the points of comparison. These gaps include a lack of systematic consistency in task design, participant selection, and evaluation metrics. Moreover, there needs to be a systematic way of analyzing the data and reporting the findings, which is also lacking. We thus propose and introduce a user protocol for KGC designed to address this challenge. Where possible, we draw and take elements from the literature we deem fit for such a protocol. The protocol, as such, allows for the comparison of languages and techniques for the RDF Mapping Language (RML) core functionality, which is covered by most of the other state-of-the-art techniques and tools. We also propose how the protocol can be amended to compare extensions (of RML). This protocol provides an important step towards a more comparable evaluation of KGC user studies.