Improved Knowledge Graph Embedding using Background Taxonomic Information

TL;DR

This paper introduces a simple yet effective method to incorporate background taxonomic information into knowledge graph embeddings, improving their expressiveness and prediction accuracy.

Contribution

It presents minimal modifications to existing models to enable the injection of taxonomic information and proves the model's full expressiveness under certain conditions.

Findings

Enhanced embedding models respect subclass and subproperty constraints.

Experimental results show improved performance on public knowledge graphs.

Model remains simple yet effective in leveraging taxonomic background information.

Abstract

Knowledge graphs are used to represent relational information in terms of triples. To enable learning about domains, embedding models, such as tensor factorization models, can be used to make predictions of new triples. Often there is background taxonomic information (in terms of subclasses and subproperties) that should also be taken into account. We show that existing fully expressive (a.k.a. universal) models cannot provably respect subclass and subproperty information. We show that minimal modifications to an existing knowledge graph completion method enables injection of taxonomic information. Moreover, we prove that our model is fully expressive, assuming a lower-bound on the size of the embeddings. Experimental results on public knowledge graphs show that despite its simplicity our approach is surprisingly effective.

Figures

Tables

Table 1. Table 1: Results for the choice of non-linearity in producing non-negative embeddings.

	WN18					FB15k
	MRR		Hit@			MRR		Hit@
Function $f$	Filter	Raw	1	3	10	Filter	Raw	1	3	10
SimplE+-Exponential	$0.866$	$0.547$	$0.829$	$0.925$	$0.897$	$0.575$	$0.248$	$0.468$	$0.640$	$0.773$
SimplE+-Logistic	$0.854$	$0.542$	$0.836$	$0.863$	$0.885$	$0.425$	$0.228$	$0.294$	$0.491$	$0.694$
SimplE+-ReLU	$0.937$	$0.575$	$0.936$	$0.938$	$0.939$	$0.725$	$0.240$	$0.658$	$0.770$	$0.841$

Table 2. Table 2: Statistics on the datasets.

Dataset	$\left|ℰ\right|$	$\left|ℛ\right|$	#train	#valid	#test
WN18	40,943	18	141,442	5,000	5,000
FB15k	14,951	1,345	483,142	50,000	59,071
Sport	1039	5	1312	-	307
Location	445	5	384	-	100

Table 3. Table 3: Relations and Rules in Sport and Location datasets.

	Relations	Subsumptions
Sport	AthleteLedSportsTeam AthletePlaysForTeam CoachesTeam OrganizationHiredPerson PersonBelongsToOrganization	$(x,AtheleLedSportsTeam,y)\to (x,AthletePlaysForTeam,y)$ $(x,AthletePlaysForTeam,y)\to (x,PersonBelongsToOrganization,y)$ $(x,CoachesTeam,y)\to (x,PersonBelongsToOrganization,y)$ $(x,OrganizationHiredPerson,y)\to (y,PersonBelongsToOrganization,x)$ $(x,PersonBelongsToOrganization,y)\to (y,OrganizationHiredPerson,x)$
Location	CapitalCityOfCountry CityLocatedInCountry CityLocatedInState StateHasCapital StateLocatedInCountry	$(x,CapitalCityOfCountry,y)\to (x,CityLocatedInCountry,y)$ $(x,StateHasCapital,y)\to (y,CityLocatedInState,x)$

Table 4. Table 4: Results on WN18 and fb15k for SimplE and SimplE + without incorporating subsumptions.

	WN18					FB15K
	MRR		Hit@			MRR		Hit@
Model	Filter	Raw	1	3	10	Filter	Raw	1	3	10
ComplEx	$0.941$	$0.587$	$0.936$	$0.945$	$0.947$	$0.692$	$0.242$	$0.599$	$0.759$	$0.840$
SimplE	$0.942$	$0.588$	$0.939$	$0.944$	$0.947$	$0.727$	$0.239$	$0.660$	$0.773$	$0.838$
SimplE+	$0.937$	$0.575$	$0.936$	$0.938$	$0.939$	$0.725$	$0.240$	$0.658$	$0.770$	$0.841$

Table 5. Table 5: Results on Sport and Location. Best results are in bold. MRR and Hit@n for n > > 1 does not make sense for logical inference.

	Sport					Location
	MRR		Hit@			MRR		Hit@
Model	Filter	Raw	1	3	10	Filter	Raw	1	3	10
Logical inference	-	-	$0.288$	-	-	-	-	$0.270$	-	-
SimplE	$0.230$	$0.174$	$0.184$	$0.234$	$0.324$	$0.190$	$0.189$	$0.130$	$0.210$	$0.315$
SimplE+	0.404	0.337	0.349	0.440	0.508	0.440	0.434	0.430	0.440	0.450

Equations

$$⟨\mathbf{x},\mathbf{y},\mathbf{z}⟩\doteq \sum _{\ell =1}^k{\mathbf{x}}_{\ell }{\mathbf{y}}_{\ell }{\mathbf{z}}_{\ell }$$

$$\mu (\mathbf{h},\mathbf{r},\mathbf{t})\doteq \sigma (\mathrm{Re}(⟨\mathbf{h},\mathbf{r},\overline{\mathbf{t}}⟩)$$

$$\mu (\mathbf{h},\mathbf{r},\mathbf{t})\doteq \sigma \left(\frac{1}{2}(⟨{\mathbf{h}}^{+},{\mathbf{r}}^{+},{\mathbf{t}}^{+}⟩+⟨{\mathbf{t}}^{-},{\mathbf{r}}^{-},{\mathbf{h}}^{-}⟩)\right).$$

$$\mathcal{L}(\{\mathbf{e}\},\{\mathbf{r}\})=-\sum _{(h,r,t)\in \mathcal{W}}\log (\mu (\mathbf{h},\mathbf{r},\mathbf{t}))$$

$$-\sum _{(h,r,t)\in {\mathcal{W}}^{\mathsf{c}}}\log (1-\mu (\mathbf{h},\mathbf{r},\mathbf{t}))+\Omega (\{\mathbf{e}\},\{\mathbf{r}\})$$

$$\mathrm{MRR}\doteq \frac{1}{2\ast |\mathcal{T}|}\sum _{(h,r,t)\in \mathcal{T}}\frac{1}{{\mathrm{rank}}_h(h,r,t)}+\frac{1}{{\mathrm{rank}}_t(h,r,t)}$$

Full text

The AI community has long noticed the importance of structure in data. While traditional machine learning techniques have been mostly focused on feature-based representations, the primary form of data in the subfield of Statistical Relational AI (StaRAI) (Getoor and Taskar, 2007; Raedt et al., 2016) is in the form of entities and relationships among them. Such entity-relationships are often in the form of $(\text{head, relationship, tail})$ triples, which can also be expressed in the form of a graph, with nodes as entities and labeled directed edges as relationships among entities. Predicting the existence, identity, and attributes of entities and their relationships are among the main goals of StaRAI.

Knowledge Graphs (KGs) are graph structured knowledge bases that store facts about the world. A large number of KGs have been created such as NELL (Carlson et al., 2010), Freebase (Bollacker et al., 2008), and Google Knowledge Vault (Dong et al., 2014). These KGs have applications in several fields including natural language processing, search, automatic question answering and recommendation systems. Since accessing and storing all the facts in the world is difficult, KGs are incomplete. The goal of link prediction for KGs – a.k.a. KG completion – is to predict the unknown links or relationships in a KG based on the existing ones. This often amounts to infer (the probability of) new triples from the existing triples.

A common approach to apply machine learning to symbolic data, such as text, graph and entity-relationships, is through embeddings. Word, sentence and paragraph embeddings (Mikolov et al., 2013; Pennington, Socher, and Manning, 2014), which vectorize words, sentences and paragraphs using context information, are widely used in a variety of natural language processing tasks from syntactic parsing to sentiment analysis. Graph embeddings (Hoff, Raftery, and Handcock, 2002; Grover and Leskovec, 2016; Perozzi, Al-Rfou, and Skiena, 2014) are used in social network analysis for link prediction and community detection.

In relational learning, embeddings for entities and relationships are used to generalize from existing data. These embeddings are often formulated in terms of tensor factorization (Nickel, Tresp, and Kriegel, 2012; Bordes et al., 2013; Trouillon et al., 2016; Kazemi and Poole, 2018c). Here, the embeddings are learned such that their interaction through (tensor-)products best predicts the (probability of the) existence of the observed triples; see (Nguyen, 2017; Wang et al., 2017) for details and discussion. Tensor factorization methods have been very successful, yet they rely on a large number of annotated triples to learn useful representations. There is often other information in ontologies which specifies the meaning of the symbols used in a knowledge base. One type of ontological information is represented in a hierarchical structure called a taxonomy. For example, a knowledge base might contain information that DJTrump, whose name is “Donald Trump” is a president, but may not contain information that he is a person, a mammal and an animal, because these are implied by taxonomic knowledge. Being told that mammals are chordates, lets us conclude that DJTrump is also a chordate, without needing to have triples specifying this about multiple mammals. We could also have information about subproperties, such as that being president is a subproperty of “managing”, which in turn is a subproperty of “interacts with”.

This paper is about combining taxonomic information in the form of subclass and subproperty (e.g., managing implies interaction) into relational embedding models. We show that existing factorization models that are fully expressive cannot reflect such constraints for all legal entity embeddings. We propose a model that is provably fully expressive and can represent such taxonomic information, and evaluate its performance on real-world datasets.

Factorization and Embedding

Let $\mathcal{E}$ represent the set of entities and $\mathcal{R}$ represent the set of relations. Let $\mathcal{W}$ be a set of triples $(h,r,t)$ that are true in the world, where $h,t\in{\mathcal{E}~{}}$ are head and tail, and $r\in{\mathcal{R}~{}}$ is the relation in the triple. We use $\mathcal{W}^{\mathsf{c}}$ to represent the triples that are false – i.e., $\mathcal{W}^{\mathsf{c}}\doteq\{(h,r,t)\in{\mathcal{E}~{}}\times{\mathcal{R}~{}}\times{\mathcal{E}~{}}\mid(h,r,t)\notin\mathcal{W}\}$. An example of a triple in $\mathcal{W}$ can be $(\text{Paris, CapitalCityOfCountry, France})$ and an example of a triple in $\mathcal{W}^{\mathsf{c}}$ can be $(\text{Paris, CapitalCityOfCountry, Germany})$. A KG $\mathcal{K}\subseteq\mathcal{W}$ is a subset of all the facts. The problem of the KG completion is to infer $\mathcal{W}$ from its subset KG. There exists a variety of methods for KG completion. Here, we consider embedding methods and in particular using tensor-factorization. For a broader review of the existing KG completion that can use background information see Related Work.

Embeddings: An embedding is a function from an entity or a relation to a vector (or sometimes higher order tensors) over a field. We use bold lower-case for vectors – that is $\mathbf{s}\in\mathds{R}^{k}$ is an embedding of an entity and $\mathbf{r}\in\mathds{R}^{l}$ is an embedding of a relation.

Taxonomies: It is common to have structure over the symbols used in the triples, see (e.g.,  Shoham, 2016). The Ontology Web Language (OWL) (Hitzler et al., 2012) defines (among many other meta-relations) subproperties and subclasses, where $p_{1}$ is a subproperty of $p_{2}$ if $\forall x,y:(x,p_{1},y)\rightarrow(x,p_{2},y)$, that is whenever $p_{1}$ is true, $p_{2}$ is also true. Classes can be defined either as a set with a class assertion (often called “type”) between an entity and a class, e.g., saying $x$ is in class $C$ using $(x,type,C)$ or in terms of the characteristic function of the class, a function that is true of element of the class. If $c$ is the characteristic function of class $C$, then $x$ is in class $c$ is written $(x,c,true)$. For representations that treat entities and properties symmetrically, the two ways to define classes are essentially the same. $C_{1}$ is a subclass of $C_{2}$ if every entity in class $C_{1}$ is in class $C_{2}$, that is, $\forall x:(x,type,C_{1})\rightarrow(x,type,C_{2})$ or $\forall x:(x,c_{1},true)\rightarrow(x,c_{2},true)$ . If we treat $true$ as an entity, then subclass can be seen as a special case of subproperty. For the rest of the paper we will refer to subsumption in terms of subproperty (and so also of subclass). A non-trivial subsumption is one which is not symmetric; $p_{1}$ is a subproperty of $p_{2}$ and there is some relations that is true of $p_{1}$ that is not true of $p_{2}$. We want the subsumption to be over all possible entities; those entities that have a legal embedding according to the representation used, not just those we know exist. Let $\mathcal{E^{*}}$ be the set of all possible entities with a legal embedding according to the representation used.

Tensor factorization: For KG completion a tensor factorization defines a function $\mu:\mathds{R}^{k}\times\mathds{R}^{l}\times\mathds{R}^{k}\to[0,1]$ that takes the embeddings $\mathbf{h}$, $\mathbf{r}$ and $\mathbf{t}$ of a triple $(h,r,t)$ as input, and generates a prediction, e.g., a probability, of the triple being true $(h,r,t)\in\mathcal{W}$. In particular, $\mu$ is often a non-linearity applied to a multi-linear function of $\mathbf{h},\mathbf{r},\mathbf{t}$. The family of methods that we study uses the following multi-linear form: Let $\mathbf{x}$, $\mathbf{y}$, and $\mathbf{z}$ be vectors of length $k$. Define $\left\langle\mathbf{x},\mathbf{y},\mathbf{z}\right\rangle$ to be the sum of their element-wise product, namely

[TABLE]

where $\mathbf{x}_{\ell}$ is the $\ell$-th element of vector $\mathbf{x}$.

Here, we are interested in creating a tensor-factorization method that is fully expressive and can incorporate background information in the form of taxonomy. A model is fully expressive if given any assignment of truth values to all triples, there exists an assignment of values to the embeddings of the entities and relations that accurately separates the triples belonging to $\mathcal{W}$ and $\mathcal{W}^{\mathsf{c}}$ using $\mu$.

ComplEx

ComplEx (Trouillon et al., 2016) defines the reconstruction function $\mu$, such that the embedding of each entity and each relation is a vector of complex numbers. Let $\mathrm{Re}(\mathbf{x})$ and $\mathrm{Im}(\mathbf{x})$ denote the real and imaginary part of a complex vector $\mathbf{x}$. In ComplEx, the probability of any triple $(h,r,t)$ is

[TABLE]

where $\sigma:\mathds{R}\to[0,1]$ is the sigmoid or logistic function, and $\overline{\mathbf{a}+i\mathbf{b}}\doteq\mathbf{a}-i\mathbf{b}$ (where $i=\sqrt{-1}$) is the element-wise conjugate of the complex vector $\mathbf{a}+i\mathbf{b}$. Note that, if the tail did not use the conjugate, the head and tail would be treated symmetrically and it could only represent symmetric relations; e.g., see DistMult in Yang et al. (2014).

Trouillon et al. (2017) prove that ComplEx is fully expressive. In particular, they prove that any assignment of ground truth can be modeled by ComplEx embeddings of length $|{\mathcal{E}~{}}||{\mathcal{R}~{}}|$. The following theorem shows that we cannot use ComplEx to enforce our prior knowledge about taxonomies.

Theorem 1**.**

*ComplEx cannot enforce non-trivial subsumption. *

Proof**.**

Assume a non-trivial subsumption so that $\forall h,t\in\mathcal{E^{*}}:(h,r,t)\rightarrow(h,s,t)$, and so $\mu(\mathbf{h},\mathbf{s},\mathbf{t})\geq\mu(\mathbf{h},\mathbf{r},\mathbf{t})$, and there are entities $a,b\in\mathcal{E^{*}}$ such that $\mu(\mathbf{a},\mathbf{s},\mathbf{b})>\mu(\mathbf{a},\mathbf{r},\mathbf{b})$. Let $a^{\prime}$ be an entity such that $\mathbf{a}^{\prime}=-\mathbf{a}$. Then $\mu(\mathbf{a}^{\prime},\mathbf{s},\mathbf{b})=1-\mu(\mathbf{a},\mathbf{s},\mathbf{b})$ and $\mu(\mathbf{a}^{\prime},\mathbf{r},\mathbf{b})=1-\mu(\mathbf{a},\mathbf{r},\mathbf{b})$, so $\mu(\mathbf{a}^{\prime},\mathbf{s},\mathbf{b})<\mu(\mathbf{a}^{\prime},\mathbf{r},\mathbf{b})$, a contradiction to the subsumption we assumed. $\blacksquare$ $\blacksquare$

Recently, Ding et al. (2018) proposed a method which they call ComplEx-NNE+AER to incorporate a weaker notion of subsumption in ComplEx. For a subsumption $\forall h,t\in\mathcal{E^{*}}:(h,r,t)\rightarrow(h,s,t)$, they suggest adding soft constraints to the loss function to encourage $\mathrm{Re}(\mathbf{r})\leq\mathrm{Re}(\mathbf{s})$ and $\mathrm{Im}(\mathbf{r})=\mathrm{Im}(\mathbf{s})$. When the constraints are satisfied, ComplEx-NNE+AER ensures $\forall h,t\in{\mathcal{E}~{}}:\mu(\mathbf{h},\mathbf{r},\mathbf{t})\leq\mu(\mathbf{h},\mathbf{s},\mathbf{t})$. This is a weaker notion than the definition in the Factorization and Embedding section which requires $\forall h,t\in{\mathcal{E}~{}}:\mu(\mathbf{h},\mathbf{r},\mathbf{t})\leq\mu(\mathbf{h},\mathbf{s},\mathbf{t})$ (that is, $\mathcal{E^{*}}$ is replaced with $\mathcal{E}$).

Theorem 2**.**

*ComplEx-NNE+AER cannot satisfy its constraints and be fully expressive if symmetry constraints are allowed. *

Proof**.**

In ComplEx a relation $r$ is symmetric for all possible entities if and only if $\mathrm{Im}(\mathbf{r})=0$ (Trouillon et al., 2016, Section 3). In order to satisfy constraints for $\forall h,t\in\mathcal{E}:(h,r,t)\rightarrow(h,s,t)$, Ding et al. (2018) assign $\mathrm{Im}(\mathbf{r})=\mathrm{Im}(\mathbf{s})$. Therefore, if relation $r$ is symmetric, it enforces relation $s$ to be symmetric too which is not generally true. As a counter example, $r$ might be the married_to relation, which is symmetric (so the $\mathrm{Im}(\textbf{married\_to})=0$), but $s$ is the knows relation, and $\forall h,t\in\mathcal{E}:(h,married\_to,t)\rightarrow(h,knows,t)$ is true in real-world, but setting the $\mathrm{Im}(\textbf{knows})$ = $\mathrm{Im}(\textbf{married\_to})$ will imply knows is symmetric, which is not true (as many people know celebrities but celebrities do not know many people). $\blacksquare$ $\blacksquare$

SimplE

SimplE (Kazemi and Poole, 2018c) achieves state-of-the-art in KG completion by considering two embeddings for each relation: one for the relation $r\in{\mathcal{R}~{}}$ itself and one for its inverse. We use $\mathbf{r}^{+}\in\mathds{R}^{k}$ to denote the “forward” embedding of $r$ and $\mathbf{r}^{-}\in\mathds{R}^{k}$ to denote the embedding of its inverse. The embedding $\mathbf{r}=[\mathbf{r}^{+},\mathbf{r}^{-}]$ for a relation is a concatenation of these two parts. Similarly, the embedding for each entity $e\in{\mathcal{E}~{}}$ has two parts: its embedding as a head $\mathbf{e}^{+}$ and as a tail $\mathbf{e}^{-}$ – that is $\mathbf{e}=[\mathbf{e}^{+},\mathbf{e}^{-}]$. Using this notation, SimplE calculates the probability of $(h,r,t)\in\mathcal{W}$ for each triple in both forward and backward directions using

[TABLE]

Kazemi and Poole (2018c) prove SimplE is fully expressive and provide a bound on the size of the embedding vectors: For any truth assignment $\mathcal{W}\subseteq{\mathcal{E}~{}}\times{\mathcal{R}~{}}\times{\mathcal{E}~{}}$, there exists a SimplE model with embedding vectors of size $\min(|{\mathcal{E}~{}}||{\mathcal{R}~{}}|,|\mathcal{W}|+1)$ that represent the assignment. The following theorem shows the limitation of SimplE when it comes to enforcing subsumption.

Theorem 3**.**

*SimplE cannot enforce non-trivial subsumptions. *

Proof**.**

Consider $\forall h,t\in\mathcal{E^{*}}:(h,r,t)\rightarrow(h,s,t)$ as a non-trivial subsumption. So we have $\mu(\mathbf{h},\mathbf{s},\mathbf{t})\geq\mu(\mathbf{h},\mathbf{r},\mathbf{t})$, and there are entities $a,b\in\mathcal{E^{*}}$ such that $\mu(\mathbf{a},\mathbf{s},\mathbf{b})>\mu(\mathbf{a},\mathbf{r},\mathbf{b})$. Let $a^{\prime}$ be an entity such that $\mathbf{a}^{\prime}=-\mathbf{a}$. Then $\mu(\mathbf{a}^{\prime},\mathbf{s},\mathbf{b})=1-\mu(\mathbf{a},\mathbf{s},\mathbf{b})$ and $\mu(\mathbf{a}^{\prime},\mathbf{r},\mathbf{b})=1-\mu(\mathbf{a},\mathbf{r},\mathbf{b})$, so $\mu(\mathbf{a}^{\prime},\mathbf{s},\mathbf{b})<\mu(\mathbf{a}^{\prime},\mathbf{r},\mathbf{b}))$ a contradiction to the subsumption we assumed. $\blacksquare$ $\blacksquare$

Neural network models

The neural network models (Socher et al., 2013; Dong et al., 2014; Santoro et al., 2017) are very flexible, and so without explicit mechanisms to enforce subsumption, they cannot be guaranteed to obey any subsumption knowledge.

Proposed Variation: SimplE*+*

In this section we propose a slight modification on SimplE so that the resulting method can enforce subsumption. The modification is restricting entity embeddings to be non-negative – that is $\mathbf{e}^{+},\mathbf{e}^{-}\geq 0\;\forall e\in{\mathcal{E}~{}}$, where the inequality is element-wise. Next we show that the resulting model is fully expressive and is able to enforce subsumption.

Theorem 4** (Expressivity).**

*For any truth assignment over entities ${\mathcal{E}~{}}$ and relations ${\mathcal{R}~{}}$ containing $|\mathcal{W}|$ true facts, there exists a SimplE+model with embeddings vectors of size $\min(|{\mathcal{E}~{}}||{\mathcal{R}~{}}|+1,|\mathcal{W}|+1)$ that represent the assignment. *

Proof**.**

Assume $r_{i}$ is the $i$-th relation in $\mathcal{R}$ and $e_{j}$ is the $j$-th entity in $\mathcal{E}$. For a vector $a$ we define $(a)_{i}$ as the $i$-th element of $a$. We define $(\mathbf{r}^{+}_{i})_{n}=1$ if $n\nonscript\mskip-4.0mu plus -2.0mu minus -4.0mu\mkern 5.0mu\mathbin{\operator@font div}\penalty 900\mkern 5.0mu\nonscript\mskip-4.0mu plus -2.0mu minus -4.0mu|\mathcal{E}|=i$ except the last element $(\mathbf{r}^{+}_{i})_{|\mathcal{E}||\mathcal{R}|}=-1$, and for each entity $s_{i}$ we define $(\mathbf{s}^{+}_{j})_{n}=1$ if $n\bmod|\mathcal{E}|=j$ or $n=|\mathcal{E}||\mathcal{R}|$ and 0 otherwise. In this setting, for each $r_{i}$ and $e_{j}$ product of $\mathbf{r}_{i}$ and $\mathbf{s}_{i}$ is [math] everywhere except for the element at $(i*|\mathcal{E}|+j)$ and the last element in the embeddings. In order for the triple $(e_{j},r_{i},e_{k})$ to hold, we define $(\mathbf{s}_{k}^{-})$ to be a vector where all elements are [math] except the ($i*|\mathcal{E}|+j$)-th element which is $2$. This proves that SimplE+is fully expressive with the bound of $|{\mathcal{E}~{}}||{\mathcal{R}~{}}|+1$ for size of the embeddings.

We use induction to prove the bound $|\mathcal{W}|+1$. Let $|\mathcal{W}|=0$ (base of induction). We can have embedding vectors of size 1 for each entity and relation, setting the value for entities to 1 and to relations to -1. Then $\left\langle\mathbf{h}^{+},\mathbf{r}^{+},\mathbf{t}^{+}\right\rangle$ + $\left\langle\mathbf{t}^{-},\mathbf{r}^{-},\mathbf{h}^{-}\right\rangle$ is negative for every entities $h$ and $t$ and relation $r$. So there exist an assignment of size 1 that represent this ground truth.

Let’s assume for any ground truth where $|\mathcal{W}|=n-1$, there exists an assignment of values to embedding vectors of size $n$ that represent the ground truth (assumption of induction). We must prove for any ground truth where $|\mathcal{W}|=n$, there exist an assignment of values to embedding vectors of size $n+1$ that represent this ground truth.

Let $(h,r,t)$ be one of the $n$ true facts. Consider a modified ground truth which is identical to the ground truth with $n$ true facts, except that $(h,r,t)$ is assigned false. The modified ground truth has $n-1$ true facts and based on the assumption of the induction, we can represent it using some embedding vectors of size $n$. Let $q=\left\langle\mathbf{h}^{+},\mathbf{r}^{+},\mathbf{t}^{+}\right\rangle+\left\langle\mathbf{t}^{-},\mathbf{r}^{-},\mathbf{h}^{-}\right\rangle$. We add an element to the end of all embedding vectors and set it to [math]. This increases the vector size to $n+1$ but does not change any scores. Then we set $\mathbf{h}$ to $1$, $\mathbf{r}$ to 1 and $\mathbf{t}$ to $q+1$. This ensure this triple is true for the new vectors, and no other probability of triple is affected. $\blacksquare$ $\blacksquare$

Theorem 5** (Subsumption).**

*SimplE+ guarantees subsumption using an inequality constraints. *

Proof**.**

Assume $\forall h,t\in\mathcal{E^{*}}:(h,r,t)\rightarrow(h,s,t)$ as a non-trivial subsumption. As legal entity embeddings in SimplE+have non-negative elements, by adding the element-wise inequality constraint $\mathbf{s}\geq\mathbf{r}$, we force $\mu(\mathbf{h},\mathbf{s},\mathbf{t})\geq\mu(\mathbf{h},\mathbf{r},\mathbf{t})$ for all $h,t\in\mathcal{E^{*}}$ which is forcing the subsumption. $\blacksquare$ $\blacksquare$

Objective Function and Training

Given the function $\mu$, that maps embeddings to the probability of a triple, ideally we would like to minimize the following regularized negative log-likelihood function:

[TABLE]

where {$\mathbf{e}$} represents entity embeddings, {$\mathbf{r}$} represents relation embeddings and $\Omega(\{\mathbf{e}\},\{\mathbf{r}\})$ is a regularization term. We use L2-regularization in our experiments. Optimizing $\mathcal{L}$ poses two challenges: I) we do not know the sets $\mathcal{W}$ and $\mathcal{W}^{\mathsf{c}}$, as the purpose of KG completion is to produce these sets in the first place; II) the number of triples (specially in $\mathcal{W}^{\mathsf{c}}$) is often too large, and for larger KGs exact calculation of these terms is often computationally unfeasible.

To address I, we use $\mathcal{K}$ as a surrogate for $\mathcal{W}$ and use its complement $\mathcal{K}^{\mathsf{c}}={\mathcal{E}~{}}\times{\mathcal{R}~{}}\times{\mathcal{E}~{}}-\mathcal{K}$ instead of $\mathcal{W}^{\mathsf{c}}$. To address the computational problem in II, we use stochastic optimization and follow the contrastive approach of Bordes et al. (2013): for each mini-batch of positive samples from KG, we produce a mini-batch of negative samples of the same size, by randomly “corrupting” the head or tail of the triple – i.e., replacing it with a random entity.

Enforcing the subsumptions

In order to enforce $\forall h,t\in\mathcal{E^{*}}:(h,r,t)\rightarrow(h,s,t)$, we add an equality constraint as $\mathbf{r}=\mathbf{s}-\delta_{r}$, where $\delta_{r}$ is a non-negative vector that specifies how $r$ differs from $s$. We learn $\delta_{r}$ for all relations $r$ that are in such a subsumption. This equality constraint guarantees the inequality constraint of Theorem 5.

Experimental Results

The objective of our empirical evaluations is two-fold: First, we want to see the practical implication of non-negativity constraints in terms of effectiveness of training and the quality of final results. Second, and more importantly, we would like to evaluate the practical benefit of incorporating prior knowledge in the form of subsumptions in sparse data regimes.

Datasets: We conducted experiments on four standard benchmarks: WN18, FB15k, Sport and Location. WN18 is a subset of Wordnet (Miller, 1995) and FB15k is a subset of Freebase (Bollacker et al., 2008). Sport and Location datasets are introduced by Wang et al. (2015), who created them using NELL (Mitchell et al., 2015). The relations in Sport and Location, along with the subsumptions, are listed in Table 3. Table 2 gives a summary of these datasets. For evaluation on WN18, FB15k, we split the existing triples in KG into the same train, validation, and test sets using the same split as (Bordes et al., 2013).

Evaluation Metrics: To evaluate different KG completion methods we need to use a train $\mathcal{N}$ and test $\mathcal{T}$ split, where $\mathcal{N}\cup\mathcal{T}=\mathcal{K}$. We use two evaluation metrics: hit@t and Mean Reciprocal Rank (MRR). Both these measures rely on the ranking of a triple in the test set $(h,r,t)\in\mathcal{T}$, obtained by corrupting the head (or the tail) of the relation with $h^{\prime}\neq h$ and estimating $\mu(h^{\prime},r,t)$. An indicator for a good KG completion method is that $(h,r,t)$ ranks high in the sorted list among corrupted triples.

Let $\mathrm{rank}_{h}(h,r,t)$ be the ranking of $\mu(h,r,t)$ among all head-corrupted relations, and let $\mathrm{rank}_{t}(h,r,t)$ denote a similar ranking with tail corruptions. MRR is the mean of the reciprocal rank:

[TABLE]

To provide a better metric, Bordes et al. (2013) suggest removing any corrupted relation that is in KG. We refer to the original definition of MRR as raw MRR and to Bordes et al. (2013)’s modified version as filtered MRR.

hit@t measures the proportion of triples in $\mathcal{T}$ that rank among top $t$ after corrupting both heads and tails.

Effect of Non-Negativity Constraints

Non-negativity has been a subject studied in various research fields. In many NLP-related tasks, non-negativity constraints are studies to learn more interpretable representations for words (Murphy, Talukdar, and Mitchell, 2012). In matrix factorization, non-negativity constraints are used to produce more coherent and independent factors (Lee and Seung, 1999). Ding et al. (2018) also proposed using non-negativity constraint to incorporate subsumption into ComplEx. We use the non-negativity constraint in SimplE+ to enforce monotonousity of probabilities as dictated by subsumption. In order to get non-negativity constraint on the embedding of entities, we simply apply an element-wise non-linearity $\phi:\mathds{R}\to\mathds{R}^{\geq 0}$ before evaluation – that is we replace $\mu(\mathbf{h},\mathbf{r},\mathbf{t})$ with $\mu(\phi(\mathbf{h}),\mathbf{r},\phi(\mathbf{t}))$.

Table 1 shows the result of SimplE*+* with for different choices of $\phi$: I) exponential $\phi(x)=e^{x}$; II) logistic $\phi(x)={(1+e^{-x})}^{-1}$; and III) rectified linear unit (ReLU) $\phi(x)=\max(x,0)$. ReLU outperforms other choices, and therefore moving forward we use ReLU for non-negativity constraints.

Next, we evaluate the effect of non-negativity constraint on the performance of the algorithm. Table 4 shows our result on WN18 and FB15k datasets. Note that this is effectively comparing SimplE+with SimplE and ComplEx, without accommodating any subsumptions. As the results indicate, this constraint does not deteriorate the model’s performance.

Sparse Relations

In this section, we study the scenario of learning relations that appear in few triples in the KG. In particular, we observe the behaviour of various methods as the amount of training triples varies. We train SimplE, SimplE+, and logical inference on fractions of the Sport training set and test them on the full test set. Logical inference refers to inferring new triples based only on the subsumptions.

Figure 1 shows the hit@1 of the three methods when they are trained on different fractions (percentages) of the training data. According to Figure 1, when training data is scarce, logical inference performs better than (or on-par with) SimplE, as SimplE does not see enough triples to be able to learn meaningful embeddings. As the amount of training data increases, SimplE starts to outperform logical inference as it can better generalize to unseen cases than pure logical inference. The gap between these two methods becomes larger as the amount of training data increases. For all tested fractions, SimplE+outperforms both SimplE and logical inference as it uses both the generalization power of SimplE and the inference power of logical rules.

In order to test the effect of incorporating taxonomical information on the number of epochs required for training to converge, we tested SimplE and SimplE+on the Sport dataset with the same set of parameters and the same initialization and plotted the loss function for each epoch. The plot in Figure 2 shows that SimplE+requires fewer epochs than SimplE to converge.

KGs with no Redundant Triples

Tensor factorization techniques rely on large amounts of annotated data. When background knowledge is available, we might expect a KG to not include redundant information. For instance if we have $(Paris,CapitalCityOfCountry,France)$ in a kg and we know $\forall h,t\in\mathcal{E^{*}}:(h,CapitalCityOfCountry,t)\rightarrow(h,CityLocatedInCountry,t)$, then the triple $(Paris,CityLocatedInCountry,France)$ is redundant. Similar to the experiment for incorporating background knowledge in Kazemi and Poole (2018c), we remove all redundant triples from the training set and compare SimplE with SimplE+and logical inference. The obtained results in Table 5 demonstrate that SimplE+outperforms SimplE and logical inference on both Sport and Location datasets with a large margin. As an example, SimplE+gains almost 90 percent and 230 percent improvement over SimplE in terms of hit@1 for Sport and Location datasets respectively. These results represent the clear advantage of SimplE+over SimplE when background taxonomic information is available.

Related Work

Incorporating background knowledge in link prediction methods has been the focus of several studies. Here, we categorize these approaches emphasizing the shortcomings that are addressed in our work; see (Nickel et al., 2016) for a review of KG embedding methods.

Soft rules There is a large family of link prediction models based on soft first-order logic rules Richardson and Domingos (2006); De Raedt, Kimmig, and Toivonen (2007); Kazemi et al. (2014). While these models can be easily integrated with background taxonomic information, they typically cannot generalize to unseen cases beyond their rules. Exceptions include Fatemi, Kazemi, and Poole (2016); Kazemi and Poole (2018b) which combine (stacked layers of) soft rules with entity embeddings, but these models have only applied to property prediction. Approaches based on path-constrained random walks (e.g., Lao and Cohen (2010)) suffer from similar limitations as they have been shown to be a subset of probabilistic logic-based models Kazemi and Poole (2018a).

Augmentation by grounding of the rules The simplest way to incorporate a set of rules in the KG is to augment the KG with their groundings (Sedghi and Sabharwal, 2018) before learning the embedding. Demeester, Rocktäschel, and Riedel (2016) address the computational inefficiency of this approach through lifted rule injection. However, in addition to being inefficient, the the resulting model does not guarantee the subsumption in the completed KG.

Augmentation through post-processing A simple approach is to augment the KG after learning the embedding using an existing method (Wang et al., 2015; Wei et al., 2015). That is, as a post processing step we can modify the output of KG completion so as to satisfy the ontological constraints. The drawback of this approach is that the background knowledge does not help learn a better representation.

Regularized embeddings Rocktäschel, Singh, and Riedel (2015) regularize the learned embeddings using first-order logic rules. In this work, every logic rule is grounded based on observations and a differentiable term is added to the loss function for every grounding. For example, grounding the rule $\forall x:human(x)\rightarrow animal(x)$ would result in a very large number of loss terms to be added to the loss function in a large KG. This method as well as other approaches in this category (e.g.,  Rocktäschel et al., 2014; Wang et al., 2015; Wang and Cohen, 2016) do not scale beyond a few entities and rules, because of the very large number of regularization terms added to the loss function (Demeester, Rocktäschel, and Riedel, 2016). Guo et al. (2018) proposed a methods for incorporating entailment into ComplEx called RUGE which models rules based on t-norm fuzzy logic, which imposes an independence assumption over the atoms. Such an independence assumption is not necessarily true, especially in the case of subsumption, e.g. in $human(x)\rightarrow animal(x)$ for which the left and the right part of the subsumption are strongly dependent. In addition to being inefficient, the resulting model of the regularized embedding approaches does not guarantee the subsumption in the completed KG.

Constrained matrix factorization Several recent works incorporate background ontologies into the embeddings learned by matrix factorization (e.g.,  Rocktäschel, Singh, and Riedel, 2015; Demeester, Rocktäschel, and Riedel, 2016). While these methods address the problems of the two categories above, they are inadequate due to the use of matrix factorization. Application of matrix factorization for KG completion (Riedel et al., 2013) learns a distinct embedding for each head-tail combination. In addition to its prohibitive memory requirement, since entities do not have their own embeddings, some regularities in the KG are ignored; for example this representation is oblivious to the fact that $(h_{i},r_{k},t_{j})$ and $(h_{l},r_{m},t_{j})$ share the same tail.

Constrained translation-based methods In translation-based methods, the relation between two entities is represented using an affine transformation, often in the form of translation. Most relevant to our work is KALE (Guo et al., 2016) that constrains the representation to accommodate logical rules, albeit after costly propositionalization. Several recent works show that a variety of existing translation-based methods are not fully expressive (Wang et al., 2017; Kazemi and Poole, 2018c), putting a severe limitation on the kinds of KGs that can be modeled using translation-based approaches.

Region based representation Gutiérrez-Basulto and Schockaert (2018) propose representing relations as convex regions in a $2k$-dimensional space, where $k$ is the length of the entity embeddings. A relation between two embeddings is deemed true if the corresponding point is in the convex region of the relation. Although this framework allows Gutiérrez-Basulto and Schockaert (2018) to incorporate a subset of existential rules by restricting the convex regions of relations, they did not propose a practical method for learning and their method is restricted to a subset of existential rules.

Conclusion and Future Work

In this paper, we proposed SimplE+, a fully expressive tensor factorization model for knowledge graph completion when background taxonomic information (in terms of subclasses and subproperties) is available. We showed that existing fully expressive models cannot provably respect subclass and subproperty information. Then we proved that by adding non-negativity constraints to entity embeddings of SimplE, a state-of-the-art tensor factorization approach, we can build a model that is not only fully expressive but also able to enforce subsumptions. Experimental results on benchmark KGs demonstrate that SimplE+is simple yet effective. On our benchmarks, SimplE+outperforms SimplE and offers a faster convergence rate when background taxonomic information is available. In future, we plan to extend SimplE+to further incorporate ontological background information, and rules such as $\forall h,t\in\mathcal{E^{*}}:(h,r,t)\land(h,s,t)\rightarrow(h,p,t)$.