Unlocking Crowdsourcing for Ontology Matching Validation

TL;DR

This paper presents a novel crowdsourcing system with domain-specific mechanisms to improve ontology matching validation, addressing challenges posed by large language models and traditional expert-based methods.

Contribution

It introduces a new crowdsourcing approach with three mechanisms to enhance validation quality and integrates human-in-the-loop validation into existing OM systems.

Findings

Effective handling of diverse user groups and annotation settings

Successful integration with existing OM systems for validation

Demonstrated system effectiveness through real-world use cases

Abstract

Recent advances in large language models (LLMs) pose new challenges for ontology matching (OM). While OM systems built on LLMs have shown remarkable capabilities in discovering more matching candidates, traditional OM validation that relies on domain experts has become overwhelming. In this study, we explore the use of crowdsourcing for OM validation and introduce a novel crowdsourcing system. We propose three domain-specific mechanisms, namely differential trustworthiness, coherence pre-filling, and time-dependent opinion, to ensure the quality of crowdsourcing for OM validation. We demonstrate that our crowdsourcing system can be integrated with existing OM systems to enable human-in-the-loop validation. The evaluation of the system also shows its effectiveness in handling diverse user groups and different annotation settings. We also discuss two real-world use cases and current limitations for improvement.