Modelling Monotonic and Non-Monotonic Attribute Dependencies with Embeddings: A Theoretical Analysis

TL;DR

This paper provides a theoretical analysis of entity embedding models, revealing their limitations in capturing certain attribute dependencies and identifying strategies capable of modeling both monotonic and non-monotonic relationships.

Contribution

It offers a theoretical perspective on the capabilities and limitations of embedding strategies for modeling attribute dependencies, including negative results and potential solutions.

Findings

Popular embedding models cannot capture basic Horn rules.

Some strategies can model both monotonic and non-monotonic dependencies.

Theoretical limitations are identified for common embedding approaches.

Abstract

During the last decade, entity embeddings have become ubiquitous in Artificial Intelligence. Such embeddings essentially serve as compact but semantically meaningful representations of the entities of interest. In most approaches, vectors are used for representing the entities themselves, as well as for representing their associated attributes. An important advantage of using attribute embeddings is that (some of the) semantic dependencies between the attributes can thus be captured. However, little is known about what kinds of semantic dependencies can be modelled in this way. The aim of this paper is to shed light on this question, focusing on settings where the embedding of an entity is obtained by pooling the embeddings of its known attributes. Our particular focus is on studying the theoretical limitations of different embedding strategies, rather than their ability to effectively learn attribute dependencies in practice. We first show a number of negative results, revealing that some of the most popular embedding models are not able to capture even basic Horn rules. However, we also find that some embedding strategies are capable, in principle, of modelling both monotonic and non-monotonic attribute dependencies.