A Comparative Study of Competency Question Elicitation Methods from Ontology Requirements
Reham Alharbi, Valentina Tamma, Terry R. Payne, Jacopo de Berardinis

TL;DR
This study empirically compares manual, pattern-based, and LLM-driven methods for formulating competency questions in ontology engineering, highlighting their differences and the potential of LLMs with refinement.
Contribution
It provides the first multi-annotator dataset of CQs from various methods and systematically compares their characteristics.
Findings
LLMs can initially elicit CQs but need refinement.
Different approaches produce CQs with varying acceptability and relevance.
The study offers insights into the strengths and limitations of each method.
Abstract
Competency Questions (CQs) are pivotal in knowledge engineering, guiding the design, validation, and testing of ontologies. A number of diverse formulation approaches have been proposed in the literature, ranging from completely manual to Large Language Model (LLM) driven ones. However, attempts to characterise the outputs of these approaches and their systematic comparison are scarce. This paper presents an empirical comparative evaluation of three distinct CQ formulation approaches: manual formulation by ontology engineers, instantiation of CQ patterns, and generation using state of the art LLMs. We generate CQs using each approach from a set of requirements for cultural heritage, and assess them across different dimensions: degree of acceptability, ambiguity, relevance, readability and complexity. Our contribution is twofold: (i) the first multi-annotator dataset of CQs generated…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
