Language Model Analysis for Ontology Subsumption Inference

TL;DR

This paper introduces OntoLAMA, a set of inference tasks and datasets to evaluate how well pre-trained language models understand ontology subsumption, revealing their limitations and potential in logical reasoning about complex concepts.

Contribution

It presents OntoLAMA, a novel framework for probing language models' knowledge of ontology subsumption, extending beyond simple triple-based knowledge bases to complex, logic-based ontologies.

Findings

Language models encode less background knowledge of subsumption inference than natural language inference.

Performance improves significantly with few-shot samples.

The study covers diverse ontologies across different domains and scales.

Abstract

Investigating whether pre-trained language models (LMs) can function as knowledge bases (KBs) has raised wide research interests recently. However, existing works focus on simple, triple-based, relational KBs, but omit more sophisticated, logic-based, conceptualised KBs such as OWL ontologies. To investigate an LM's knowledge of ontologies, we propose OntoLAMA, a set of inference-based probing tasks and datasets from ontology subsumption axioms involving both atomic and complex concepts. We conduct extensive experiments on ontologies of different domains and scales, and our results demonstrate that LMs encode relatively less background knowledge of Subsumption Inference (SI) than traditional Natural Language Inference (NLI) but can improve on SI significantly when a small number of samples are given. We will open-source our code and datasets.