Can Large Language Models Understand DL-Lite Ontologies? An Empirical Study

TL;DR

This study empirically evaluates large language models' ability to understand DL-Lite ontologies, revealing their strengths in syntax and semantics but limitations with transitivity and large ABoxes.

Contribution

It provides the first comprehensive analysis of LLMs' understanding of DL-Lite ontologies across multiple tasks, highlighting both capabilities and shortcomings.

Findings

LLMs understand formal syntax of DL-Lite.

LLMs grasp model-theoretic semantics of concepts and roles.

LLMs struggle with transitivity and large ABoxes.

Abstract

Large language models (LLMs) have shown significant achievements in solving a wide range of tasks. Recently, LLMs' capability to store, retrieve and infer with symbolic knowledge has drawn a great deal of attention, showing their potential to understand structured information. However, it is not yet known whether LLMs can understand Description Logic (DL) ontologies. In this work, we empirically analyze the LLMs' capability of understanding DL-Lite ontologies covering 6 representative tasks from syntactic and semantic aspects. With extensive experiments, we demonstrate both the effectiveness and limitations of LLMs in understanding DL-Lite ontologies. We find that LLMs can understand formal syntax and model-theoretic semantics of concepts and roles. However, LLMs struggle with understanding TBox NI transitivity and handling ontologies with large ABoxes. We hope that our experiments and analyses provide more insights into LLMs and inspire to build more faithful knowledge engineering solutions.