Large Language Models Assisting Ontology Evaluation

TL;DR

This paper introduces OE-Assist, a framework using large language models to automate and assist ontology evaluation through competency question verification, reducing manual effort and improving accuracy.

Contribution

The work presents the first systematic investigation of LLM-assisted ontology evaluation, including a dataset, evaluation of LLM effectiveness, and a Protégé plugin for CQ verification.

Findings

LLM-based CQ verification performs comparably to average users.

The dataset includes 1,393 CQs with ontologies and stories.

Automated evaluation reduces manual effort in ontology assessment.

Abstract

Ontology evaluation through functional requirements, such as testing via competency question (CQ) verification, is a well-established yet costly, labour-intensive, and error-prone endeavour, even for ontology engineering experts. In this work, we introduce OE-Assist, a novel framework designed to assist ontology evaluation through automated and semi-automated CQ verification. By presenting and leveraging a dataset of 1,393 CQs paired with corresponding ontologies and ontology stories, our contributions present, to our knowledge, the first systematic investigation into large language model (LLM)-assisted ontology evaluation, and include: (i) evaluating the effectiveness of a LLM-based approach for automatically performing CQ verification against a manually created gold standard, and (ii) developing and assessing an LLM-powered framework to assist CQ verification with Prot\'eg\'e, by providing suggestions. We found that automated LLM-based evaluation with o1-preview and o3-mini perform at a similar level to the average user's performance.