LOUC: Leave-One-Out-Calibration Measure for Analyzing Human Matcher Performance

TL;DR

This paper introduces LOUC, a novel calibration-based measure for analyzing human matcher performance in schema matching, enabling detailed behavioral insights to improve annotation quality.

Contribution

It proposes a new calibration measure for evaluating human matchers, advancing understanding of human performance in schema matching tasks.

Findings

LOUC effectively analyzes human matcher behavior

Calibration correlates with matching accuracy

Potential to enhance annotation quality and algorithms

Abstract

Schema matching is a core data integration task, focusing on identifying correspondences among attributes of multiple schemata. Numerous algorithmic approaches were suggested for schema matching over the years, aiming at solving the task with as little human involvement as possible. Yet, humans are still required in the loop -- to validate algorithms and to produce ground truth data for algorithms to be trained against. In recent years, a new research direction investigates the capabilities and behavior of humans while performing matching tasks. Previous works utilized this knowledge to predict, and even improve, the performance of human matchers. In this work, we continue this line of research by suggesting a novel measure to evaluate the performance of human matchers, based on calibration, a common meta-cognition measure. The proposed measure enables detailed analysis of various factors of the behavior of human matchers and their relation to human performance. Such analysis can be further utilized to develop heuristics and methods to better asses and improve the annotation quality.