OAEI-LLM: A Benchmark Dataset for Understanding Large Language Model Hallucinations in Ontology Matching

TL;DR

This paper introduces OAEI-LLM, a benchmark dataset designed to evaluate and understand hallucinations of large language models in ontology matching tasks, addressing a critical need for specialized evaluation tools.

Contribution

The paper presents the creation of the OAEI-LLM dataset, an extended benchmark for assessing LLM hallucinations in ontology matching, including methodology and potential applications.

Findings

OAEI-LLM effectively captures LLM hallucinations in OM tasks.

The dataset enables systematic evaluation of LLM reliability in ontology matching.

Potential use cases for improving LLM-based ontology matching are demonstrated.

Abstract

Hallucinations of large language models (LLMs) commonly occur in domain-specific downstream tasks, with no exception in ontology matching (OM). The prevalence of using LLMs for OM raises the need for benchmarks to better understand LLM hallucinations. The OAEI-LLM dataset is an extended version of the Ontology Alignment Evaluation Initiative (OAEI) datasets that evaluate LLM-specific hallucinations in OM tasks. We outline the methodology used in dataset construction and schema extension, and provide examples of potential use cases.