Bench4KE: Benchmarking Automated Competency Question Generation

TL;DR

Bench4KE introduces a standardized benchmarking system for evaluating automated Competency Question generation tools using real-world datasets and similarity metrics, enabling fair comparison and fostering progress in KE automation research.

Contribution

It provides an extensible, standardized benchmarking framework with curated datasets and evaluation metrics for CQ generation tools, addressing the lack of evaluation consistency in KE automation.

Findings

Six CQ generation systems evaluated, establishing a baseline.

Benchmarking system facilitates fair comparison of KE automation tools.

Code and datasets are publicly available for reproducibility.

Abstract

The availability of Large Language Models (LLMs) presents a unique opportunity to reinvigorate research on Knowledge Engineering (KE) automation. This trend is already evident in recent efforts developing LLM-based methods and tools for the automatic generation of Competency Questions (CQs), natural language questions used by ontology engineers to define the functional requirements of an ontology. However, the evaluation of these tools lacks standardization. This undermines the methodological rigor and hinders the replication and comparison of results. To address this gap, we introduce Bench4KE, an extensible API-based benchmarking system for KE automation. The presented release focuses on evaluating tools that generate CQs automatically. Bench4KE provides a curated gold standard consisting of CQ datasets from 17 real-world ontology engineering projects and uses a suite of similarity metrics to assess the quality of the CQs generated. We present a comparative analysis of 6 recent CQ generation systems, which are based on LLMs, establishing a baseline for future research. Bench4KE is also designed to accommodate additional KE automation tasks, such as SPARQL query generation, ontology testing and drafting. Code and datasets are publicly available under the Apache 2.0 license.