Measuring Expert Performance at Manually Classifying Domain Entities under Upper Ontology Classes

TL;DR

This study assesses how accurately domain experts classify entities under upper ontology classes, revealing the task's difficulty and highlighting the need for improved methodologies in ontology integration.

Contribution

It provides the first empirical measurement of expert performance in classifying entities under upper ontology classes, emphasizing the task's complexity and inconsistency.

Findings

Manual classification is very challenging and often inconsistent.

Experts struggle to correctly classify entities under upper ontology classes.

The study highlights the need for better methodological support in ontology integration.

Abstract

Classifying entities in domain ontologies under upper ontology classes is a recommended task in ontology engineering to facilitate semantic interoperability and modelling consistency. Integrating upper ontologies this way is difficult and, despite emerging automated methods, remains a largely manual task. Little is known about how well experts perform at upper ontology integration. To develop methodological and tool support, we first need to understand how well experts do this task. We designed a study to measure the performance of human experts at manually classifying classes in a general knowledge domain ontology with entities in the Basic Formal Ontology (BFO), an upper ontology used widely in the biomedical domain. We conclude that manually classifying domain entities under upper ontology classes is indeed very difficult to do correctly. Given the importance of the task and the high degree of inconsistent classifications we encountered, we further conclude that it is necessary to improve the methodological framework surrounding the manual integration of domain and upper ontologies.