Language Models as Ontology Encoders

TL;DR

This paper introduces OnT, a novel ontology embedding method that combines pretrained language models with geometric hyperbolic space modeling to better incorporate textual labels and logical structures, improving inference and prediction tasks.

Contribution

The paper presents a new ontology embedding approach, OnT, that effectively integrates textual information and logical relationships using hyperbolic geometry and pretrained language models.

Findings

OnT outperforms existing baselines in ontology prediction and inference tasks.

OnT demonstrates strong transfer learning capabilities.

OnT effectively constructs ontologies from real-world data like SNOMED CT.

Abstract

OWL (Web Ontology Language) ontologies which are able to formally represent complex knowledge and support semantic reasoning have been widely adopted across various domains such as healthcare and bioinformatics. Recently, ontology embeddings have gained wide attention due to its potential to infer plausible new knowledge and approximate complex reasoning. However, existing methods face notable limitations: geometric model-based embeddings typically overlook valuable textual information, resulting in suboptimal performance, while the approaches that incorporate text, which are often based on language models, fail to preserve the logical structure. In this work, we propose a new ontology embedding method OnT, which tunes a Pretrained Language Model (PLM) via geometric modeling in a hyperbolic space for effectively incorporating textual labels and simultaneously preserving class hierarchies and other logical relationships of Description Logic EL. Extensive experiments on four real-world ontologies show that OnT consistently outperforms the baselines including the state-of-the-art across both tasks of prediction and inference of axioms. OnT also demonstrates strong potential in real-world applications, indicated by its robust transfer learning abilities and effectiveness in real cases of constructing a new ontology from SNOMED CT. Data and code are available at https://github.com/HuiYang1997/OnT.