A Pattern to Align Them All: Integrating Different Modalities to Define Multi-Modal Entities

TL;DR

This paper introduces a new ontology design pattern to unify and integrate diverse modalities in Multi-Modal Knowledge Graphs, enhancing reasoning and application across various domains.

Contribution

It proposes an abstract model that separates entity semantics from their physical representations, aiding the harmonization of existing multi-modal ontologies.

Findings

Facilitates integration of multi-modal ontologies

Supports reasoning with diverse sensory data

Enhances cross-domain applications

Abstract

The ability to reason with and integrate different sensory inputs is the foundation underpinning human intelligence and it is the reason for the growing interest in modelling multi-modal information within Knowledge Graphs. Multi-Modal Knowledge Graphs extend traditional Knowledge Graphs by associating an entity with its possible modal representations, including text, images, audio, and videos, all of which are used to convey the semantics of the entity. Despite the increasing attention that Multi-Modal Knowledge Graphs have received, there is a lack of consensus about the definitions and modelling of modalities, whose definition is often determined by application domains. In this paper, we propose a novel ontology design pattern that captures the separation of concerns between an entity (and the information it conveys), whose semantics can have different manifestations across different media, and its realisation in terms of a physical information entity. By introducing this abstract model, we aim to facilitate the harmonisation and integration of different existing multi-modal ontologies which is crucial for many intelligent applications across different domains spanning from medicine to digital humanities.