The Knowledge Graph Track at OAEI -- Gold Standards, Baselines, and the Golden Hammer Bias

TL;DR

This paper discusses the design and initial results of the OAEI Knowledge Graph track, highlighting the discovery of a bias affecting ontology matching tools, called the golden hammer bias.

Contribution

It introduces the Knowledge Graph track at OAEI, compares gold standard creation strategies, and reveals the golden hammer bias affecting matching tools.

Findings

Tools suffer from the golden hammer bias.

Analysis of initial track results.

Insights into gold standard creation strategies.

Abstract

The Ontology Alignment Evaluation Initiative (OAEI) is an annual evaluation of ontology matching tools. In 2018, we have started the Knowledge Graph track, whose goal is to evaluate the simultaneous matching of entities and schemas of large-scale knowledge graphs. In this paper, we discuss the design of the track and two different strategies of gold standard creation. We analyze results and experiences obtained in first editions of the track, and, by revealing a hidden task, we show that all tools submitted to the track (and probably also to other tracks) suffer from a bias which we name the golden hammer bias.