From human experts to machines: An LLM supported approach to ontology and knowledge graph construction

TL;DR

This paper presents a semi-automated pipeline leveraging open-source LLMs to construct ontologies and knowledge graphs with minimal human input, demonstrated on scholarly publications about deep learning.

Contribution

It introduces a novel semi-automatic approach combining LLMs and a judge LLM for constructing and evaluating KGs with reduced human effort.

Findings

LLMs can automate parts of ontology and KG construction.

The pipeline effectively creates a KG on deep learning topics.

Human-in-the-loop is still recommended for quality assurance.

Abstract

The conventional process of building Ontologies and Knowledge Graphs (KGs) heavily relies on human domain experts to define entities and relationship types, establish hierarchies, maintain relevance to the domain, fill the ABox (or populate with instances), and ensure data quality (including amongst others accuracy and completeness). On the other hand, Large Language Models (LLMs) have recently gained popularity for their ability to understand and generate human-like natural language, offering promising ways to automate aspects of this process. This work explores the (semi-)automatic construction of KGs facilitated by open-source LLMs. Our pipeline involves formulating competency questions (CQs), developing an ontology (TBox) based on these CQs, constructing KGs using the developed ontology, and evaluating the resultant KG with minimal to no involvement of human experts. We showcase the feasibility of our semi-automated pipeline by creating a KG on deep learning methodologies by exploiting scholarly publications. To evaluate the answers generated via Retrieval-Augmented-Generation (RAG) as well as the KG concepts automatically extracted using LLMs, we design a judge LLM, which rates the generated content based on ground truth. Our findings suggest that employing LLMs could potentially reduce the human effort involved in the construction of KGs, although a human-in-the-loop approach is recommended to evaluate automatically generated KGs.