A Comparative Study of Competency Question Elicitation Methods from Ontology Requirements

TL;DR

This study empirically compares manual, pattern-based, and LLM-driven methods for formulating competency questions in ontology engineering, highlighting their differences and the potential of LLMs with refinement.

Contribution

It provides the first multi-annotator dataset of CQs from various methods and systematically compares their characteristics.

Findings

LLMs can initially elicit CQs but need refinement.

Different approaches produce CQs with varying acceptability and relevance.

The study offers insights into the strengths and limitations of each method.

Abstract

Competency Questions (CQs) are pivotal in knowledge engineering, guiding the design, validation, and testing of ontologies. A number of diverse formulation approaches have been proposed in the literature, ranging from completely manual to Large Language Model (LLM) driven ones. However, attempts to characterise the outputs of these approaches and their systematic comparison are scarce. This paper presents an empirical comparative evaluation of three distinct CQ formulation approaches: manual formulation by ontology engineers, instantiation of CQ patterns, and generation using state of the art LLMs. We generate CQs using each approach from a set of requirements for cultural heritage, and assess them across different dimensions: degree of acceptability, ambiguity, relevance, readability and complexity. Our contribution is twofold: (i) the first multi-annotator dataset of CQs generated from the same source using different methods; and (ii) a systematic comparison of the characteristics of the CQs resulting from each approach. Our study shows that different CQ generation approaches have different characteristics and that LLMs can be used as a way to initially elicit CQs, however these are sensitive to the model used to generate CQs and they generally require a further refinement step before they can be used to model requirements.