Knowledge Graph Embedding Bi-Vector Models for Symmetric Relation

Jinkui Yao; Lianghua Xu

arXiv:1905.09557·cs.AI·May 24, 2019

Knowledge Graph Embedding Bi-Vector Models for Symmetric Relation

Jinkui Yao, Lianghua Xu

PDF

Open Access

TL;DR

This paper introduces bi-vector models for knowledge graph embeddings that better handle symmetric relations by representing them as vector pairs, addressing a key limitation in existing models and improving reasoning tasks.

Contribution

The paper proposes a novel bi-vector embedding approach for symmetric relations, enhancing the modeling of symmetry in knowledge graphs.

Findings

01

Bi-vector models outperform baseline models in symmetric relation tasks.

02

Generated benchmark datasets validate the effectiveness of the proposed models.

03

Models show significant improvement in link prediction accuracy for symmetric relations.

Abstract

Knowledge graph embedding (KGE) models have been proposed to improve the performance of knowledge graph reasoning. However, there is a general phenomenon in most of KGEs, as the training progresses, the symmetric relations tend to zero vector, if the symmetric triples ratio is high enough in the dataset. This phenomenon causes subsequent tasks, e.g. link prediction etc., of symmetric relations to fail. The root cause of the problem is that KGEs do not utilize the semantic information of symmetric relations. We propose KGE bi-vector models, which represent the symmetric relations as vector pair, significantly increasing the processing capability of the symmetry relations. We generate the benchmark datasets based on FB15k and WN18 by completing the symmetric relation triples to verify models. The experiment results of our models clearly affirm the effectiveness and superiority of our…

Tables4

Table 1. Table 1: Statistics of several popular datasets. | ℰ | ℰ |\mathcal{E}| is the number of entity, and | ℛ | ℛ |\mathcal{R}| is the number of relations, train/test/valid is the number of train/test/valid set. S Y M ‡ 𝑆 𝑌 superscript 𝑀 ‡ SYM^{\ddagger} is number of symmetric triple, S Y M † 𝑆 𝑌 superscript 𝑀 † SYM^{\dagger} is number of complement symmetric triple, A L L 𝐴 𝐿 𝐿 ALL is number of triple in dataset, A L L + S Y M † 𝐴 𝐿 𝐿 𝑆 𝑌 superscript 𝑀 † ALL+SYM^{\dagger} is number of triple in dataset after complement, S Y M ‡ A L L @ % 𝑆 𝑌 superscript 𝑀 ‡ 𝐴 𝐿 𝐿 percent @ \frac{SYM^{\ddagger}}{ALL}@\% is percentage of symmetric triples in the train/test/valid set, S Y M ‡ + 2 S Y M † A L L + S Y M † @ % 𝑆 𝑌 superscript 𝑀 ‡ 2 𝑆 𝑌 superscript 𝑀 † 𝐴 𝐿 𝐿 𝑆 𝑌 superscript 𝑀 † percent @ \frac{SYM^{\ddagger}+2SYM^{\dagger}}{ALL+SYM^{\dagger}}@\% is percentage of symmetric triples in the train/test/valid set after complement.

Dataset	$\| ℰ \|$	$\| ℛ \|$	train/test/valid	$\frac{S Y M^{‡}}{A L L} @ %$	$\frac{S Y M^{‡} + 2 S Y M^{†}}{A L L + S Y M^{†}} @ %$
FB15k	14,951	1,345	483,142/50,000/59,071	7.15/0.94/0.744	8.69/8.41/8.34
FB15k-237	14,541	237	272,115/17,535/20,466	12.48/1.44/1.13	14.97/2.65/2.58
FB13	75,043	13	316,232/5,908/23,733	1.31/0.00/0.00	1.42/0.00/0.00
WN18	40,943	18	141,442/5,000/5,000	20.97/0.52/0.72	22.38/19.07/19.01
WN11	38,696	11	112,581/2,609/10,544	1.41/0.06/0.00	1.54/0.20/0.08
WN18RR	40,943	11	86,835/3,134/3,034	34.15/0.83/1.19	36.05/27.38/27.98

Table 2. Table 2: Symmetric relation examples in FB15k and WN18. SYM is the number of symmetric relations, ALL is number of relations and S Y M A L L 𝑆 𝑌 𝑀 𝐴 𝐿 𝐿 \frac{SYM}{ALL} is the proportion of the symmetric relations in the total number of relations.

Dataset	Relation	SYM	ALL	$\frac{S Y M}{A L L}$
FB15k	/military/military_combatant/force_deployments/…/combatant	78	84	0.929
	/base/fight/crime_type/p…/crime/criminal_conviction/guilty_of	20	21	0.952
	/base/twinnedtowns/twinned_town/…/town_twinning/twinned_towns	20	21	0.952
	/base/contractbridge/…/bridge_tournament_standings/second_place	18	19	0.947
	/sports/sports_position/…/sports-_team_roster/position	108	127	0.850
WN18	_derivationally_related_form	27694	29716	0.931
	_verb_group	1060	1139	0.931
	_similar_to	74	81	0.914
	_also_see	830	1300	0.638

Table 3. Table 3: Circle triple test result.

Model	Train Dataset	Test Dataset	MR	MRR	H10	H3	H1
TransE	FB15k-SYM	FB15k-test-circle	1.000	1.000	1.000	1.000	1.000
TransH	FB15k-SYM	FB15k-test-circle	1.000	1.000	1.000	1.000	1.000
TransD	FB15k-SYM	FB15k-test-circle	1.000	1.000	1.000	1.000	1.000
TransE	WN18-SYM	WN18-test-circle	1.000	1.000	1.000	1.000	1.000
TransH	WN18-SYM	WN18-test-circle	1.000	1.000	1.000	1.000	1.000
TransD	WN18-SYM	WN18-test-circle	1.000	1.000	1.000	1.000	1.000

Table 4. Table 4: Experimental result

	MR	MRR	H10	H3	H1	MR	MRR	H10	H3	H1
	FB15k-SYM					WN18-SYM
TransE	66	0.490	0.683	0.461	0.206	493	0.371	0.711	0.544	0.087
TransE-SYM	51	0.534	0.772	0.598	0.329	467	0.485	0.836	0.705	0.246
TransH	80	0.380	0.747	0.539	0.162	688	0.426	0.926	0.828	0.026
TransH-SYM	49	0.432	0.784	0.612	0.344	601	0.577	0.931	0.845	0.120
TransD	185	0.265	0.519	0.297	0.148	711	0.416	0.928	0.787	0.145
TransD-SYM	72	0.642	0.774	0.543	0.335	210	0.886	0.941	0.866	0.374

Equations24

f_{r}(h,t)=\big{\|}h+r-t\big{\|}_{L_{n}}.

f_{r}(h,t)=\big{\|}h+r-t\big{\|}_{L_{n}}.

f_{r}(h,t)=\big{\|}(h-\omega_{r}^{\top}h\omega_{r})+r-(t-\omega_{r}^{\top}t\omega_{r})\big{\|}_{L_{n}}.

f_{r}(h,t)=\big{\|}(h-\omega_{r}^{\top}h\omega_{r})+r-(t-\omega_{r}^{\top}t\omega_{r})\big{\|}_{L_{n}}.

f_{r}(h,t)=\big{\|}M_{rh}h+r-M_{rh}t\big{\|}_{L_{n}}.

f_{r}(h,t)=\big{\|}M_{rh}h+r-M_{rh}t\big{\|}_{L_{n}}.

e_{T r u m p} + r_{s p o u se} = e_{M e l ania}

e_{T r u m p} + r_{s p o u se} = e_{M e l ania}

e_{M e l ania} + r_{s p o u se} = e_{T r u m p}

e_{M e l ania} + r_{s p o u se} = e_{T r u m p}

e_{T r u m p} + e_{M e l ania} + 2 r_{s p o u se} = e_{M e l ania} + e_{T r u m p}

e_{T r u m p} + e_{M e l ania} + 2 r_{s p o u se} = e_{M e l ania} + e_{T r u m p}

r_{s p o u se} = 0

r_{s p o u se} = 0

f_{r_{s}} (h, t) = min (f_{r_{s}^{+}} (h, t), f_{r_{s}^{-}} (h, t))

f_{r_{s}} (h, t) = min (f_{r_{s}^{+}} (h, t), f_{r_{s}^{-}} (h, t))

\displaystyle\left\{\begin{array}[]{c}f_{r^{+}_{s}}(h,t)=\big{\|}h+r^{+}_{s}-t\big{\|}_{L_{n}}\\ f_{r^{-}_{s}}(h,t)=\big{\|}h+r^{-}_{s}-t\big{\|}_{L_{n}}\end{array}\right.

\displaystyle\left\{\begin{array}[]{c}f_{r^{+}_{s}}(h,t)=\big{\|}h+r^{+}_{s}-t\big{\|}_{L_{n}}\\ f_{r^{-}_{s}}(h,t)=\big{\|}h+r^{-}_{s}-t\big{\|}_{L_{n}}\end{array}\right.

L = (h, r_{s}, t) \in Π_{r_{s}} \sum (h^{'}, r_{s}, t^{'}) \in Π_{r_{s}}^{'} \sum [λ + f_{r_{s}} (h, t) - f_{r_{s}} (h^{'}, t^{'})]_{+} .

L = (h, r_{s}, t) \in Π_{r_{s}} \sum (h^{'}, r_{s}, t^{'}) \in Π_{r_{s}}^{'} \sum [λ + f_{r_{s}} (h, t) - f_{r_{s}} (h^{'}, t^{'})]_{+} .

\displaystyle\left\{\begin{array}[]{l}f_{r^{+}_{s}}(h,t)=\big{\|}(h-\omega_{r^{+}_{s}}^{\top}h\omega_{r^{+}_{s}})+r^{+}_{s}-(t-\omega_{r^{+}_{s}}^{\top}t\omega_{r^{+}_{s}})\big{\|}_{L_{n}}\\ f_{r^{-}_{s}}(h,t)=\big{\|}(h-\omega_{r^{-}_{s}}^{\top}h\omega_{r^{-}_{s}})+r^{-}_{s}-(t-\omega_{r^{-}_{s}}^{\top}t\omega_{r^{-}_{s}})\big{\|}_{L_{n}}\\ f_{r_{s}}(h,t)=\min(f_{r^{+}_{s}}(h,t),f_{r^{-}_{s}}(h,t))\end{array}\right.

\displaystyle\left\{\begin{array}[]{l}f_{r^{+}_{s}}(h,t)=\big{\|}(h-\omega_{r^{+}_{s}}^{\top}h\omega_{r^{+}_{s}})+r^{+}_{s}-(t-\omega_{r^{+}_{s}}^{\top}t\omega_{r^{+}_{s}})\big{\|}_{L_{n}}\\ f_{r^{-}_{s}}(h,t)=\big{\|}(h-\omega_{r^{-}_{s}}^{\top}h\omega_{r^{-}_{s}})+r^{-}_{s}-(t-\omega_{r^{-}_{s}}^{\top}t\omega_{r^{-}_{s}})\big{\|}_{L_{n}}\\ f_{r_{s}}(h,t)=\min(f_{r^{+}_{s}}(h,t),f_{r^{-}_{s}}(h,t))\end{array}\right.

\displaystyle\left\{\begin{array}[]{l}f_{r^{+}_{s}}(h,t)=\big{\|}M_{r^{+}_{s}h}h+r^{+}_{s}-M_{r^{+}_{s}t}t\big{\|}_{L_{n}}\\ f_{r^{-}_{s}}(h,t)=\big{\|}M_{r^{-}_{s}h}h+r^{-}_{s}-M_{r^{-}_{s}t}t\big{\|}_{L_{n}}\\ f_{r_{s}}(h,t)=\min(f_{r^{+}_{s}}(h,t),f_{r^{-}_{s}}(h,t))\end{array}\right.

\displaystyle\left\{\begin{array}[]{l}f_{r^{+}_{s}}(h,t)=\big{\|}M_{r^{+}_{s}h}h+r^{+}_{s}-M_{r^{+}_{s}t}t\big{\|}_{L_{n}}\\ f_{r^{-}_{s}}(h,t)=\big{\|}M_{r^{-}_{s}h}h+r^{-}_{s}-M_{r^{-}_{s}t}t\big{\|}_{L_{n}}\\ f_{r_{s}}(h,t)=\min(f_{r^{+}_{s}}(h,t),f_{r^{-}_{s}}(h,t))\end{array}\right.

Peer Reviews

No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.

Videos

No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.

Taxonomy

TopicsAdvanced Graph Neural Networks · Topic Modeling · Data Quality and Management

Full text

11footnotetext: J. Yao (✉)

Jiangnan Institute of Computing Technology, WuXi, 214000, China

e-mail: [email protected]: L. Xu

Jiangnan Institute of Computing Technology, WuXi, 214000, China

Knowledge Graph Embedding Bi-Vector Models for Symmetric Relation

Jinkui Yao1

Lianghua Xu2

Abstract

Knowledge graph embedding (KGE) models have been proposed to improve the performance of knowledge graph reasoning. However, there is a general phenomenon in most of KGEs, as the training progresses, the symmetric relations tend to zero vector, if the symmetric triples ratio is high enough in the dataset. This phenomenon causes subsequent tasks, e.g. link prediction etc., of symmetric relations to fail. The root cause of the problem is that KGEs do not utilize the semantic information of symmetric relations. We propose KGE bi-vector models, which represent the symmetric relations as vector pair, significantly increasing the processing capability of the symmetry relations. We generate the benchmark datasets based on FB15k and WN18 by completing the symmetric relation triples to verify models. The experiment results of our models clearly affirm the effectiveness and superiority of our models against baseline.

Keywords:

k

nowledge graph embedding, symmetry relation, bi-vector models

1 Introduction

The knowledge graph, a structured knowledge base, represents world’s truth in a form that computer can easily process. As the basis of question answering and knowledge inference, etc., the knowledge graph has received extensive attention from academia and industry.

In recent years, knowledge graph reasoning has made significant progress. There are two main branches, logical reasoning and representation learning, each with its own advantages and disadvantages. Logical reasoning based on the rigorous mathematical foundation is difficult to solve the computational bottleneck of the combinatorial explosion. Knowledge representation learning based on statistics has attracted more attention because of the development of machine learning and deep learning at present, but it is limited by the incompleteness and the scale of the knowledge base.

Usually, each fact of the knowledge graph is represented by a triple $(h,r,t)$ , where $h$ and $t$ are the head entity and the tail entity, respectively, and $r$ is the relation between them.

For example, the triple $(Trump,\ spouse,\ Melania)$ means that Trump’s spouse is Melania, in which Trump is the head entity, the spouse is the relation, and Melania is the tail entity. Semantically, relation $spouse$ is symmetric, shown in figure1, $(Trump,\ spouse,\ Melania)$ and $(Melania,\ spouse,\ Trump)$ simultaneously hold.

KGE aims to embed the entities and relations into low-dimensional real vectors, and then learns the representations of them. TransE [1] is the earliest KGE model and has derived a series of models called Trans series models or Trans models. Most of Trans models based on vector addition calculation, which are difficult to apply well in symmetric relations.

We propose bi-vector models extended the Trans models for symmetric relations. Different from the Trans models using a single vector to represent the entity or relation, We adopt bi-vector to represent symmetric relation. The score functions of the two subvectors are calculated separately. With the increase of training epochs, the two subvectors are separated step by step. And then, models can distinguish the two directions of the symmetric relation

Two benchmark datasets, FB15k-SYM and WN18-SYM construced by us for running bi-vector models on them The experimental results show that our method can effectively improve the triple prediction accuracy of symmetry relations. The main contributions of this paper as follow.

We propose bi-vector models which improve the prediction accuracy of symmetric relations. 2. 2.

The symmetric semantic information of relations is combined with KGE, which is a new research method of knowledge graph reasoning. 3. 3.

We run the model on the extended benchmark datasets and verify the effectiveness and advantages of the models.

2 Related Works

We extend three popular KGE models, TransE, TransH [2] and TransD [3], using bi-vector. Therefore, we firstly introduce these.

2.1 TransE, TransH and TransD

•TransE

, the first KGE model proposed, regards relation $r$ as the translation from entity $h$ to $t$ . Entity $t$ should be in the nearest neighborhood of $h+r$ . The score function is defined as

[TABLE]

Where $L_{n}$ is usually as $L_{1}$ norm or $L_{2}$ norm. TransE can slove 1-1 relations effectively, but it is not suitable for handling 1-n, n-1 and n-n relations.

•TransH

projects entities $h$ and $t$ into the hyperplane which relation $r$ located. TransH calculates $h_{\bot}=h-\omega_{r}^{\top}h\omega_{r}$ , $t_{\bot}=t-\omega_{r}^{\top}t\omega_{r}$ before calculating score function,

[TABLE]

Where $L_{n}$ is usually as $L_{2}$ norm. TransH is more accurate than TransE in terms of recognition rate of 1-n, n-1 and n-n relations.

•TransD

believes that combinations of entities and relations can distinguish the relation more finely. The combination of entity $h$ and relation $r$ correspondences association matrix $M_{r}h$ . The calculation of score function uses the product of entity and association matrix, form as $h_{\bot}=M_{rh}h$ , $t_{\bot}=M_{rh}t$ . The score function is defined as

[TABLE]

Where $L_{n}$ is usually as $L_{2}$ norm.

2.2 Other Models

•Translation based methods

. In addition to TransE(H,D) that we have already mentioned, translation based methods cover the following models. TransR [4] build entity and relation embedding independent spaces, in which, entities $h,t\in\mathbb{R}^{k}$ , and relation $r\in\mathbb{R}^{d}$ . A projection matrix $M_{r}\in\mathbb{R}^{k\times d}$ has been set, and the score funcion is defined as $f_{r}(h,t)=\big{\|}M_{r}h+r-M_{r}t\big{\|}_{2}^{2}.$ TransSparse [5] set two separate relation sparse matrices $M_{r}^{h}(\theta_{r}^{h})$ and $M_{r}^{t}(\theta_{r}^{t})$ to deal with the issue of sparse data. The score function is defined as $\big{\|}M_{r}^{h}(\theta_{r}^{h})h+r-M_{r}^{t}(\theta_{r}^{t})t\big{\|}_{L_{n}}$ .TransF reduces the cost of calculation of relation projection by modeling subspaces of projection matrices, and the score function is defined as $f_{r}(h,t))=\big{\|}(\sum_{i=1}^{s}\alpha_{r}^{(i)}U^{(i)}+I)h+r+(\sum_{i=1}^{s}\beta_{r}^{(i)}V^{(i)}+I)t\big{\|}_{L_{n}}$ , where $s\in\mathbb{R}$ , ${U^{(i)}},{V^{(i)}}\in\mathbb{R}^{d_{e}\times d_{r}}$ , $\alpha_{r}^{(i)}U^{(i)}$ and $\beta_{r}^{(i)}V^{(i)}+I)$ are the corresponding coefficients of ${U^{(i)}}$ and ${V^{(i)}}$ .

•Tensor based methods

. DistMult [6] adopts a relation-specific diagonal matrix $M_{r}$ to represents the characteristics of a relation. The score function $f_{r}(h,t)=hM_{r}t$ is a bilinear function, which score of positive triples should be higher than negative triples. HolE [7] employs circular correlations by holographic to create compositional representations, and has advantages of computation efficiency and representing scalability. RESCAL [8] adopt tensor factorization to estimate relation axis.ComplEX [9] embed the entities and relation to complex space, then computes loss vaule.

•Other related methods

. SE[10] defines two relation-specific matrices for $h,t$ , i.e. $M_{r,1},M_{r,2}$ , and defines the score function as $f_{r}(h,t)=\big{\|}M_{h,r}h-M_{t,r}t\big{\|}_{1}$ . There are many other KGE models try to try to use various embedding methods, such as Neural Tensor Network (NTN)[11] , Semantic Matching Energy (SME)[12], SLM, TransA, lppTransD, etc.

However, these works did not utilize the semantic information of relations properties. We believe that the semantic information of the relations properties are of value and can improve the performance of the KGE models.

3 Methodology

In order to overcome the lack of support for symmetric relations in KGE, we made the following efforts. First of all we describe the defects of Trans models in handling symmetric relations, and analyze the causes of it. Then, we propose three new models that extends the Trans models to improve the performance of handling symmetry relations in KGE, which are named TransE-SYM, TransH-SYM and TransD-SYM. Finally, we give the definition of the loss functions for these models.

3.1 Problems and causes

Knowledge graph can be represented as a set of ordered triples of entities and relations. Each triple in Knowledge graph is essentially a binary relation, which have the properties of symmetry, anti-symmetric, reflexive, anti-reflexive and transitive properties. This paper focuses on the relation’s properties of symmetry. In graph, symmetric relation have two directed edges in opposite directions.

KGE represents each relation, including symmetric relation, as a low-dimensional real vector. However, a single vector cannot represent two opposite directions.

We take TransE as an example to illustrate the problem of symmetric relations. TransE learns the embedding feature from equation $h+r=t$ when triplets $(h,r,t)$ holds. TransE’s scoring function is defined as $f_{r}(h,t)=\big{\|}h+r-t\big{\|}_{L_{n}}$ . When the function $f_{r}(h,t)=0$ , it means $h+r=t$ .

Assuming that there is a symmetric relation $r_{s}$ and triple $(h,r_{s},t)$ in $KG$ , then $h+r_{s}=t$ , ie $r_{s}=t-h$ . Since $r_{s}$ is symmetric, then the symmetric triple $(t,r_{s},h)$ should hold too, satisfying $t+r_{s}=h$ , ie $r_{s}=h-t$ .

Obviously, if both $r_{s}=t-h$ and $r_{s}=h-t$ are correct, if and only if $r_{s}$ is an additive identity of vector, ie $r_{s}\equiv\vec{0}$ , the conclusion contradicts with the conditions of TransE model.

Taking the symmetric relation $spouse$ as an example, shown in figure1. When the fact $(Trump,spouse,Melania)$ holds, the fact $(Melania,spouse,Trump)$ holds too. let $e_{Trump}$ , $e_{Melania}$ and $e_{Melania}$ denote entities Melania, Trump and relation spouse, respectively. Then,

[TABLE]

let Equation(4) + Equation(5),

[TABLE]

we have

[TABLE]

According to the KGE preset, the relations $r_{spouse}$ should be a non-zero real vector, and Equation (6) contradicts with the condition. The root cause of the above problem is that the symmetric relation is represented by single vector, and the single vector cannot express semantic bifurcation of symmetric relation.

3.2 Our Method

Aiming at these problems, bi-vector models for symmetric relation are presented in this study.

Knowledge graph $KG$ , $KG{}=\{(h,r,t)\}\subseteq E\times R\times E$ , Where $E$ and $R$ are entities set and relations set, respectively.

Symmetric relation $r_{s}$ , if $h$ and $t$ are entities of knowledge graph $KG$ , $r_{s}$ is the relation of $KG$ , and $(h,r_{s},t)\subseteq KG$ , $(t,r_{s},h)\subseteq KG$ , then relation $r_{s}$ is symmetric relation.

Different from most of KGE models, which represent entities and relations as single vector, we represent the symmetric relation $r_{s}$ as a bi-vector with two subvectors, $r^{+}_{s}$ and $r^{-}_{s}$ . Then, in each epoch of learning, the score functions of the two subvectors are calculated, and the better score is selected as the current result. Let $f_{r_{s}}(h,t)$ be the score function of the Trans series model, as show in Equation(7)

[TABLE]

We have extended three different Trans models, which differ in their respective score functions. In TransE, score function is $f_{r_{s}}(h,t)=\big{\|}h+r_{s}-t\big{\|}_{L_{n}}$ , where ${L_{n}}$ is L1 norm or L2 norm, and the score functions of subvectors are shown as Equation array(10),

[TABLE]

$f_{r_{s}}(h,t)$ and $r_{s}$ should be substituted into the following loss function,

[TABLE]

where $\lambda>0$ denotes the margin of hyperplane, and $[x]_{+}$ denotes $\max(x,0)$ . Similarly, the score function of the TransH model is shown in Equation array (15).

[TABLE]

The score function of the TransH model is shown in Equation array (19).

[TABLE]

The loss functions of them are calculated according to Equation (11).

4 Experiments and results

4.1 Dataset analysis and preprocessing

In this study, we compared and analyzed the commonly used knowledge graph embedding benchmark data sets FB15k, FB15k-237, FB13, WN18, WN11 and WN18RR. FB15k, FB15k-237 and FB13 are extracted from Freebase[13], which is a large-scale common sense knowledge base provided the general facts of the world. Freebase was acquired by Google and is still under maintenance. WN18, WN11 and WN18R aextract from WordNet [14] and provide semantic knowledge of words.

We count the ratio of the symmetric relations in the data set shown in the table 1. It can be seen that the proportion of symmetric data of the WN18 and FB15k data set are relatively high.

The proportion of symmetric data for relation $r$ is denoted as $\zeta_{r}$ by the paper. We regard $r$ as symmetric relation When $\zeta_{r}$ exceeds the threshold111In this paper, the threshold is set to 0.5..

As shown in the table2, in WN18, the relation $\_verb\_group$ has 1139 triples, of which 1060 are symmetric triples, and the ratio of symmetric triples is about 0.93. Semantically, the relation $\_verb\_group$ is the meaning of verb grouping, which is obviously a symmetric relation. From the perspective of data distribution, the symmetry rate of the relation $\_verb\_group$ is 0.93, and we believe it is symmetrical.

In order to simplify the problem, in this paper, symmetry is only judged by data distribution.We complement the missing symmetric triples in dataset of the symmetric relation. A more formal description is, if relation $r_{s}$ in knowledge graph $KG$ is symmetric, for $\forall(h,r_{s},t)\in KG$ , if $(t,r_{s},h)\notin KG$ and then $KG=KG\cup(t,r,h)$ .

4.2 Benchmarks

In order to show the superiority of our models, we compare the following benchmark KGE models.

•TransE

is the most widely used KGE model, also the earliest proposed KGE model.

•TransH

projects h and t to the hyperplane where r located, to solve the relations of 1-n, n-1 and n-n.

•TransD

uses the entity-relation matrix to obtain a more fine-grained distinction of realtion.

4.3 Verification problem

In order to verify the problem of the Trans models described in Section 3.1, We have designed the following experiments, the steps are as follows.

Training Trans models. We train the TransE, TransH and TransD models on the datasets which are completed symmetric triples in Section 4.1. 2. 2.

**Constructing test dataset.**We randomly selected symmetric relations and entities in FB15k and WN18 to construct test sets. Each test set contains 10,000 symmetric triples named FB15k-test-circle and WN18-test-circle. The form of triples in test sets is $(e,r_{s},e)$ , where $r_{s}$ and $e$ are respectively symmetric relation and any entity. The triple example is as follows,

$(05451384,\_derivationally\_related\_form,05451384)$ ,

$(04958634,\_verb\_group,04958634)$ . 3. 3.

Experimental results. According to Section 3.1, if the symmetric triple is true, the relation tends to zero. We run the test sets on models and the experimental results are shown in Table 3. Almost all randomly generated triples is true. These models completely fail in dealing with all of symmetric relations.

4.4 Result of Experiment.

Three bi-vector Trans models named TransE-SYM, TransH-SYM and TransD-SYM proposed by us. Experimental code implementation reference open source project OpenKE[15]. These models run on datasets completed symmetric relation and get good results. The experimental results are shown in Table 4. Bi-vector models are superior to the original model in indicators of the link prediction task.

5 Conclusion

This paper introduces symmetry semantics into KGE models, and points out the defect of the state-of-the-art KGE models learning symmetric relations. Bi-vector models proposed by us can improve the situation of low recognition rate of symmetric relations in Trans models.

Bibliography15

The reference list from the paper itself. Each links out to its DOI / PubMed record.

1[1] Antoine Bordes, Nicolas Usunier, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In NIPS.
2[2] Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In AAAI.
3[3] Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Knowledge graph embedding via dynamic mapping matrix. In ACL.
4[4] Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In AAAI.
5[5] Guoliang Ji, Kang Liu, Shizhu He, and Jun Zhao. (2016). Knowledge graph completion with adaptive sparse transfer matrix. Thirtieth Aaai Conference on Artificial Intelligence.
6[6] Bishan Yang, Wentau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In ICLR.
7[7] Maximilian Nickel, Lorenzo Rosasco, and Tomaso Poggio. 2016. Holographic embeddings of knowledge graphs. In AAAI.
8[8] Nickel, M., Tresp, V., Kriegel, H. P. (2011, June). A Three-Way Model for Collective Learning on Multi-Relational Data. In ICML (Vol. 11, pp. 809-816).