BLINKG: A Benchmark for LLM-Integrated Knowledge Graph Generation

TL;DR

This paper introduces BLINKG, a benchmark to evaluate how effectively large language models can assist in constructing knowledge graphs from diverse data sources, highlighting current capabilities and limitations.

Contribution

The paper presents BLINKG, a standardized benchmark for assessing LLMs in knowledge graph generation, and provides an extensive evaluation of state-of-the-art models.

Findings

LLMs show promising results in KG construction tasks.

Performance decreases in complex scenarios.

BLINKG helps identify current limitations of LLMs in this domain.

Abstract

Generating Knowledge Graphs (KGs) remains one of the most time-consuming and labor-intensive tasks for knowledge engineers, as they need to identify semantic equivalences between input data sources and ontology terms. While declarative solutions (e.g., RML, SPARQL-Anything) have helped to generalize this process, aligning input schema elements with ontology terms still involves intricate transformations and requires considerable manual effort. With the advent of Large Language Models (LLMs), there is growing interest in leveraging their capabilities to assist KG engineers. Although some studies have explored using LLMs to automate KG construction, there is still no standardized framework for assessing how effectively they establish correspondences between data schemes and ontology concepts. Therefore, in this paper, we propose BLINKG, a benchmark designed to evaluate the mapping capabilities of LLMs in constructing KGs from heterogeneous data sources. The benchmark includes a set of scenarios with increasing complexity, based on real-world use cases. We conduct an extensive experimental evaluation of several stateof-the-art LLMs using BLINK and observe that they already offer promising solutions. However, their performance remains limited in complex scenarios. Thanks to this benchmark, we can already assess the current capabilities of LLMs for KG construction. Additionally, we define a set of requirements for achieving (semi)automated (LLM-driven) KG construction, opening new research lines in this area.