Quantifying Similarity between Relations with Fact Distribution

TL;DR

This paper presents a simple neural network-based method to measure relation similarity in knowledge bases, correlating well with human judgments and improving relation detection and classification tasks.

Contribution

It introduces a sampling-based approach to approximate relation similarity via divergence of conditional distributions, enhancing relation detection and classification.

Findings

Significant correlation with human judgments

Effective detection of redundant relations

Improved relation classification accuracy

Abstract

We introduce a conceptually simple and effective method to quantify the similarity between relations in knowledge bases. Specifically, our approach is based on the divergence between the conditional probability distributions over entity pairs. In this paper, these distributions are parameterized by a very simple neural network. Although computing the exact similarity is in-tractable, we provide a sampling-based method to get a good approximation. We empirically show the outputs of our approach significantly correlate with human judgments. By applying our method to various tasks, we also find that (1) our approach could effectively detect redundant relations extracted by open information extraction (Open IE) models, that (2) even the most competitive models for relational classification still make mistakes among very similar relations, and that (3) our approach could be incorporated into negative sampling and softmax classification to alleviate these mistakes. The source code and experiment details of this paper can be obtained from https://github.com/thunlp/relation-similarity.

Figures

Tables

Table 1. Table 1: An illustration of the errors made by relation extraction models. The sentence contains obvious patterns indicating the two persons are siblings, but the model predicts it as parents. We introduce an approach to measure the similarity between relations. Our result shows “siblings” is the second most similar one to “parents”. By applying this approach, we could analyze the errors made by models, and help reduce errors.

Sentence	The crisis didn’t influence his two daughters OBJ and SUBJ.
Correct	per:siblings
Predicted	per:parents
Similarity Rank	2

Table 2. Table 2: Methods to define a similarity function with different types of relation representations

Relation Representation	Method	Similarity Quantification
Vectors	TransE Bordes et al. (2013)	$S(r_1,r_2)=\exp \left({𝒓}_1^{\top }{𝒓}_2/{\parallel {𝒓}_1\parallel }_2{\parallel {𝒓}_2\parallel }_2\right)$
Vectors	DistMult Yang et al. (2015)	$S(r_1,r_2)=\exp \left({𝒓}_1^{\top }{𝒓}_2/{\parallel {𝒓}_1\parallel }_2{\parallel {𝒓}_2\parallel }_2\right)$
Matrices	RESCAL Nickel et al. (2011)	$S(r_1,r_2)=\exp \left({\parallel M_{r_1}-M_{r_2}\parallel }_F\right)$
Angles	RotatE Sun et al. (2019)	$S(r_1,r_2)=\exp \left(-{\sum }_{i=1}^n{\left|{𝒓}_{1,i}-{𝒓}_{2,i}\right|}_1\right)$
Probability Distribution	Ours	equation 6

Table 3. Table 3: Statistics of the triple sets used in this paper.

Triple Set	$\left|ℛ\right|$	$\left|ℰ\right|$	#Fact	Section
Wikidata	188	112,946	426,067	§ 5 and § 6.1
ReVerb Extractions	3,736	194,556	266,645	§ 6.2
FB15K	1,345	14,951	483,142	§ 7.1 and § 8
TACRED	42	29,943	68,124	§ 7.2 and § 9

Table 4. Table 4: The experiment results on the toy dataset show that our metric based on probability distribution significantly outperforms other relation similarity metrics.

Method	P	R	$F_1$
Vectors (TransE)	0.28	0.14	0.18
Vectors (DistMult)	0.44	0.41	0.42
Angles	0.48	0.43	0.45
Ours	0.65	0.50	0.57

Table 5. Table 5: Improvement of using similarity in softmax-margin loss.

	Model	P	R	$F_1$
Traditional	Patterns	86.9	23.2	36.6
	LR	73.5	49.9	59.4
Neural	CNN	75.6	47.5	58.3
	CNN-PE	70.3	54.2	61.2
	SDP-LSTM Xu et al. (2015)	66.3	52.7	58.7
	LSTM	65.7	59.9	62.7
	PA-LSTM Zhang et al. (2017)	65.7	64.5	65.1
Neural+Ours	PA-LSTM (Softmax-Margin Loss)	68.5	64.7	66.6

Equations

$${\tilde{P}}_{\theta }(h,t\mid r)\triangleq \exp F_{\theta }(h,r;t)$$

$$u_{{\theta }_1}(h;r)$$

$$u_{{\theta }_2}(t;h,r)$$

$${\tilde{u}}_{{\theta }_1}(h;r)$$

$${\tilde{u}}_{{\theta }_2}(t;h,r)$$

$${\theta }^{\ast }$$

$$=\underset{\theta }{\mathrm{argmin}}\sum _{(h,r,t)\in G}-\log P_{\theta }(h,t\mid r),$$

$$P_{{\theta }^{\ast }}(h,t\mid r)=\exp (u_{{\theta }_1}(h;r)+u_{{\theta }_2}(t;h,r)).$$

$$\displaystyle S(r_{1},r_{2})=g\Big{(}$$

$$\displaystyle D_{\mathrm{KL}}\left(\left.\left.P_{\theta^{*}}(\,h,t\mid r_{2}\,)\right|\right|P_{\theta^{*}}(\,h,t\mid r_{1}\,)\right)\Big{)},$$

$$D_{\mathrm{KL}}\left(P_{{\theta }^{\ast }}(h,t\mid r_1)||P_{{\theta }^{\ast }}(h,t\mid r_2)\right)$$

$$={\mathbb{E}}_{h,t\sim P_{{\theta }^{\ast }}(h,t\mid r_1)}\log \frac{P_{{\theta }^{\ast }}(h,t\mid r_1)}{P_{{\theta }^{\ast }}(h,t\mid r_2)}$$

$$D_{\mathrm{KL}}\left(P_{{\theta }^{\ast }}(h,t\mid r_1)||P_{{\theta }^{\ast }}(h,t\mid r_2)\right)$$

$$=$$

$$\approx$$

$$Recall={\mathbb{E}}_{x\sim U}\mathbb{I}[\hat{f}(x)=1],$$

$$Recall\approx \sum _{i=1}^n\mathbb{I}[\hat{f}(x_i)=1]{\hat{w}}_i,$$

$$Precision={\mathbb{E}}_{x\sim U^{\prime }}\mathbb{I}[f(x)=1],$$

$$Precision\approx \frac{1}{n}\sum _{i=1}^n\mathbb{I}[f(x_i)=1],$$

$$\mathcal{L}=\sum _{(h,r,t)\in \mathcal{D}}[d(\mathbf{h}+\mathbf{r},\mathbf{t})-d({\mathbf{h}}^{\prime }+{\mathbf{r}}^{\prime },{\mathbf{t}}^{\prime })+\gamma {]}_{+},$$

$$p=\frac{S(r,r^{\prime }{)}^{1/\alpha }}{{\sum }_{r^{\prime \prime }\in \mathcal{R}\backslash \{r\}}S(r,r^{\prime \prime }{)}^{1/\alpha }},$$

$$\mathcal{L}=$$

$$\log \sum _{r\in \mathcal{R}(x^{(i)})}\exp \{{\theta }^{T}f(x^{(i)},r)+\text{cost}(r^{(i)},r)\},$$

$$Recall$$

$$=\sum _x\frac{\mathbb{I}[f(x)=1\wedge \hat{f}(x)=1]}{{\sum }_{x^{\prime }}\mathbb{I}[f(x^{\prime })=1]}$$

$$=\sum _x\frac{\mathbb{I}[f(x)=1]\mathbb{I}[\hat{f}(x)=1]}{{\sum }_{x^{\prime }}\mathbb{I}[f(x^{\prime })=1]}$$

$$=\sum _x\frac{\mathbb{I}[f(x)=1]}{{\sum }_{x^{\prime }}\mathbb{I}[f(x^{\prime })=1]}\mathbb{I}[\hat{f}(x)=1]$$

$$=\sum _x{P}_U(x)\mathbb{I}[\hat{f}(x)=1]$$

$$={\mathbb{E}}_{x\sim U}\mathbb{I}[\hat{f}(x)=1]$$

$$Recall$$

$${\mathbb{E}}_{x\sim q}\mathbb{I}[\hat{f}(x)=1]\frac{P_U(x)}{q(x)}$$

$$\approx \frac{{\sum }_{i=1}^n\mathbb{I}[\hat{f}(x_i)=1]P_U(x_i)/q(x_i)}{{\sum }_{i=1}^n{P}_U(x_i)/q(x_i)}$$

$$=\frac{{\sum }_{i=1}^n\mathbb{I}[\hat{f}(x_i)=1]w_i}{{\sum }_{i=1}^n{w}_i}(w_i=\frac{\mathbb{I}[f(x_i)=1]}{\tilde{q}(x_i)})$$

$$=\sum _{i=1}^n\mathbb{I}[\hat{f}(x_i)=1]{\hat{w}}_i,$$

$$p=\frac{\alpha }{\alpha +n+1}$$

$$p=\frac{n_k}{\alpha +n+1}$$

$$U=\alpha U_{all}+(1-\alpha )U_{type},$$

$$p=\{\begin{array}{ll}\frac{1+ϵ}{N}& r^{\prime }\in \mathcal{T}(r)\\ \frac{1}{N}& \text{otherwise}\end{array}$$

Full text

We empirically show the outputs of our approach significantly correlate with human judgments. By applying our method to various tasks, we also find that (1) our approach could effectively detect redundant relations extracted by open information extraction (Open IE) models, that (2) even the most competitive models for relational classification still make mistakes among very similar relations, and that (3) our approach could be incorporated into negative sampling and softmax classification to alleviate these mistakes. The source code and experiment details of this paper can be obtained from https://github.com/thunlp/relation-similarity.

1 Introduction

††footnotetext: Author contributions: Hao Zhu designed the research; Weize Chen prepared the data, and organized data annotation; Hao Zhu and Xu Han designed the experiments; Weize Chen performed the experiments; Hao Zhu, Weize Chen and Xu Han wrote the paper; Zhiyuan Liu and Maosong Sun proofread the paper. Zhiyuan Liu is the corresponding author.

Relations111Sometimes relations are also named properties., representing various types of connections between entities or arguments, are the core of expressing relational facts in most general knowledge bases (KBs) Suchanek et al. (2007); Bollacker et al. (2008). Hence, identifying relations is a crucial problem for several information extraction tasks. Although considerable effort has been devoted to these tasks, some nuances between similar relations are still overlooked, (Table 1 shows an example); on the other hand, some distinct surface forms carrying the same relational semantics are mistaken as different relations. These severe problems motivate us to quantify the similarity between relations in a more effective and robust method.

In this paper, we introduce an adaptive and general framework for measuring similarity of the pairs of relations. Suppose for each relation $r$, we have obtained a conditional distribution, $P(h,t\mid r)$ ($h,t\in\mathcal{E}$ are head and tail entities, and $r\in\mathcal{R}$ is a relation), over all head-tail entity pairs given $r$. We could quantify similarity between a pair of relations by the divergence between the conditional probability distributions given these relations. In this paper, this conditional probability is given by a simple feed-forward neural network, which can capture the dependencies between entities conditioned on specific relations. Despite its simplicity, the proposed network is expected to cover various facts, even if the facts are not used for training, owing to the good generalizability of neural networks. For example, our network will assign a fact a higher probability if it is “logical”: e.g., the network might prefer an athlete has the same nationality as same as his/her national team rather than other nations.

Intuitively, two similar relations should have similar conditional distributions over head-tail entity pairs $P(\,h,t\mid r\,)$, e.g., the entity pairs associated with be trade to and play for are most likely to be athletes and their clubs, whereas those associated with live in are often people and locations. In this paper, we evaluate the similarity between relations based on their conditional distributions over entity pairs. Specifically, we adopt Kullback–Leibler (KL) divergence of both directions as the metric. However, computing exact KL requires iterating over the whole entity pair space $\mathcal{E}\times\mathcal{E}$, which is quite intractable. Therefore, we further provide a sampling-based method to approximate the similarity score over the entity pair space for computational efficiency.

Besides developing a framework for assessing the similarity between relations, our second contribution is that we have done a survey of applications. We present experiments and analysis aimed at answering five questions:

(1) How well does the computed similarity score correlate with human judgment about the similarity between relations? How does our approach compare to other possible approaches based on other kinds of relation embeddings to define a similarity? (§ 3.4 and § 5)

(2) Open IE models inevitably extract many redundant relations. How can our approach help reduce such redundancy? (§ 6)

(3) To which extent, quantitatively, does best relational classification models make errors among similar relations? (§ 7)

(4) Could similarity be used in a heuristic method to enhance negative sampling for relation prediction? (§ 8)

(5) Could similarity be used as an adaptive margin in softmax-margin training method for relation extraction? (§ 9)

Finally, we conclude with a discussion of valid extensions to our method and other possible applications.

2 Learning Head-Tail Distribution

Just as introduced in § 1, we quantify the similarity between relations by their corresponding head-tail entity pair distributions. Consider the typical case that we have got numbers of facts, but they are still sparse among all facts in the real world. How could we obtain a well-generalized distribution over the whole space of possible triples beyond the training facts? This section proposes a method to parameterize such a distribution.

2.1 Formal Definition of Fact Distribution

A fact is a triple $(h,r,t)\in\mathcal{E}\times\mathcal{R}\times\mathcal{E}$, where $h$ and $t$ are called head and tail entities, $r$ is the relation connecting them, $\mathcal{E}$ and $\mathcal{R}$ are the sets of entities and relations respectively. We consider a score function $F_{\theta}:\mathcal{E}\times\mathcal{R}\times\mathcal{E}\rightarrow\mathbb{R}$ maps all triples to a scalar value. As a special case, the function can be factorized into the sum of two parts: $F_{\theta}(\,h,t;r\,)\triangleq u_{\theta_{1}}(h;r)+u_{\theta_{2}}(t;h,r)$. We use $F_{\theta}$ to define the unnormalized probability.

[TABLE]

for every triple $(\,h,r,t\,)$. The real parameter $\theta$ can be adjusted to obtain difference distributions over facts.

In this paper, we only consider locally normalized version of $F_{\theta}$:

[TABLE]

where $\tilde{u}_{\theta_{1}}$ and $\tilde{u}_{\theta_{2}}$ are directly parameterized by feed-forward neural networks. Through local normalization, $\tilde{P}_{\theta}(\,h,t\mid r\,)$ is naturally a valid probability distribution, as the partition function $\sum_{h,t}\exp F_{\theta}(\,h,t;r\,)=1$. Therefore, $P_{\theta}(\,h,t\mid r\,)=\tilde{P}_{\theta}(\,h,t\mid r\,)$.

2.2 Neural architecture design

Here we introduce our special design of neural networks. For the first part and the second part, we implement the scoring functions introduced in equation 2 as

[TABLE]

where each $\mathtt{MLP}_{\theta}$ represents a multi-layer perceptron composed of layers like $\bm{y}=\mathtt{relu}(\bm{W}\bm{x}+\bm{b})$, $\bm{h},\bm{r}$, $\bm{t}$ are embeddings of $h,r$, $t$, and $\theta$ includes weights and biases in all layers.

2.3 Training

Now we discuss the method to perform training. In this paper, we consider joint training. By minimizing the loss function, we compute the model parameters $\theta^{*}$:

[TABLE]

where $G\subset\mathcal{E}\times\mathcal{R}\times\mathcal{E}$ is a set of triples.222In our applications, the set of triples could be a knowledge base or a set of triples in the training set etc. The whole set of parameters, $\theta=\{\theta_{1},\theta_{2},\{\bm{e},\forall e\in\mathcal{E}\},\{\bm{r},\forall r\in\mathcal{R}\}\}$. We train these parameters by Adam optimizer Kingma and Ba (2014). Training details are shown in Appendix C.

3 Quantifying Similarity

So far, we have talked about how to use neural networks to approximate the natural distribution of facts. The center topic of our paper, quantifying similarity, will be discussed in detail in this section.

3.1 Relations as Distributions

In this paper, we provide a probability view of relations by representing relation $r$ as a probability distribution $P_{\theta^{*}}(\,h,t\mid r\,)$. After training the neural network on a given set of triples, the model is expected to generalize well on the whole $\mathcal{E}\times\mathcal{R}\times\mathcal{E}$ space.

Note that it is very easy to calculate $P_{\theta^{*}}(\,h,t\mid r\,)$ in our model thanks to local normalization (equation 2). Therefore, we can compute it by

[TABLE]

3.2 Defining Similarity

As the basis of our definition, we hypothesize that the similarity between $P_{\theta^{*}}(\,h,t\mid r\,)$ reflects the similarity between relations.333§ 5 provides empirical results to corroborate this hypothesis. For example, if the conditional distributions of two relations put mass on similar entity pairs, the two relations should be quite similar. If they emphasize different ones, the two should have some differences in meaning.

Formally, we define the similarity between two relations as a function of the divergence between the distributions of corresponding head-tail entity pairs:

[TABLE]

where $D_{\mathrm{KL}}\left(\left.\left.\cdot\right|\right|\cdot\right)$ denotes Kullback–Leibler divergence,

[TABLE]

vice versa, and function $g(\cdot,\cdot)$ is a symmetrical function. To keep the coherence between semantic meaning of “similarity” and our definition, $g$ should be a monotonically decreasing function. Through this paper, we choose to use an exponential family444We view KL divergences as energy functions. composed with max function, i.e., $g(x,y)=e^{-\max(x,y)}$. Note that by taking both sides of KL divergence into account, our definition incorporates both the entity pairs with high probability in $r_{1}$ and $r_{2}$. Intuitively, if $P_{\theta^{*}}(\,h,t\mid r_{1}\,)$ mainly distributes on a proportion of entities pairs that $P_{\theta^{*}}(\,h,t\mid r_{2}\,)$ emphasizes, $r_{1}$ is only hyponymy of $r_{2}$. Considering both sides of KL divergence could help model yield more comprehensive consideration. We will talk about the advantage of this method in detail in § 3.4.

3.3 Calculating Similarity

Just as introduced in § 1, it is intractable to compute similarity exactly, as involving $\mathcal{O}(|\mathcal{E}|^{2})$ computation. Hence, we consider the monte-carlo approximation:

[TABLE]

where $\mathcal{S}$ is a list of entity pairs sampled from $P_{\theta^{*}}(\,h,t\mid r_{1}\,)$. We use sequential sampling555Sampling $h$ and $t$ at the same time requires $\mathcal{O}(|\mathcal{E}|^{2})$ computation, while sequential sampling requires only $\mathcal{O}(|\mathcal{E}|)$ computation. to gain $\mathcal{S}$, which means we first sample $h$ given $r$ from $u_{\theta_{1}}(h;r)$, and then sample $t$ given $h$ and $r$ from $u_{\theta_{2}}(t;h,r)$.666It seems to be a non-symmetrical method, and sampling from the mixture of both forward and backward should yield a better result. Surprisingly, in practice, sampling from single direction works just as well as from both directions.

3.4 Relationship with other metrics

Previous work proposed various methods for representing relations as vectors Bordes et al. (2013); Yang et al. (2015), as matrices Nickel et al. (2011), even as angles Sun et al. (2019), etc. Based on each of these representations, one could easily define various similarity quantification methods.777Taking the widely used vector representations as an example, we can define the similarity between relations based on cosine distance, dot product distance, L1/L2 distance, etc. We show in Table 2 the best one of them in each category of relation presentation.

Here we provide two intuitive reasons for using our proposed probability-based similarity: (1) the capacity of a single fixed-size representation is limited — some details about the fact distribution is lost during embedding; (2) directly comparing distributions yields a better interpretability — you can not know about how two relations are different given two relation embeddings, but our model helps you study the detailed differences between probabilities on every entity pair. footnote 8 provides an example. Although the two relations talk about the same topic, they have different meanings. TransE embeds them as vectors the closest to each other, while our model can capture the distinction between the distributions corresponds to the two relations, which could be directly noticed from the figure.

4 Dataset Construction

We show the statistics of the dataset we use in Table 3, and the construction procedures will be introduced in this section.

4.1 Wikidata

In Wikidata Vrandečić and Krötzsch (2014), facts can be described as (Head item/property, Property, Tail item/property). To construct a dataset suitable for our task, we only consider the facts whose head entity and tail entity are both items. We first choose the most common 202 relations and 120000 entities from Wikidata as our initial data. Considering that the facts containing the two most frequently appearing relations (P2860: cites, and P31: instance of) occupy half of the initial data, we drop the two relations to downsize the dataset and make the dataset more balanced. Finally, we keep the triples whose head and tail both come from the selected 120000 entities as well as its relation comes from the remaining 200 relations.

4.2 ReVerb Extractions

ReVerb Fader et al. (2011) is a program that automatically identifies and extracts binary relationships from English sentences. We use the extractions from running ReVerb on Wikipedia999http://reverb.cs.washington.edu/. We only keep the relations appear more than 10 times and their corresponding triples to construct our dataset.

4.3 FB15K and TACRED

FB15K Bordes et al. (2013) is a subset of freebase. TACRED Zhang et al. (2017) is a large supervised relation extraction dataset obtained via crowdsourcing. We directly use these two dataset, no extra processing steps were applied.

5 Human Judgments

Following Miller and Charles (1991); Resnik (1999) and the vast amount of previous work on semantic similarity, we ask nine undergraduate subjects to assess the similarity of 360 pairs of relations from a subset of Wikidata Vrandečić and Krötzsch (2014)101010Wikidata provides detailed descriptions to properties (relations), which could help subjects understand the relations better. that are chosen to cover from high to low levels of similarity. In our experiment, subjects were asked to rate an integer similarity score from 0 (no similarity) to 4 (perfectly the same)111111The detailed instruction is attached in the Appendix F. for each pair. The inter-subject correlation, estimated by leaving-one-out method Weiss and Kulikowski (1991), is r = $0.763$, standard deviation = $0.060$. This important reference value (marked in Figure 2) could be seen as the highest expected performance for machines Resnik (1999).

To get baselines for comparison, we consider other possible methods to define similarity functions, as shown in Table 2. We compute the correlation between these methods and human judgment scores. As the models we have chosen are the ones work best in knowledge base completion, we do expect the similarity quantification approaches based on them could measure some degree of similarity. As shown in Figure 2, the three baseline models could achieve moderate ($0.1\text{--}0.5$) positive correlation. On the other hand, our model shows a stronger correlation ($0.63$) with human judgment, indicating that considering the probability over whole entity pair space helps to gain a similarity closer to human judgments. These results provide evidence for our claim raised in § 3.2.

6 Redundant Relation Removal

Open IE extracts concise token patterns from plain text to represent various relations between entities, e.g.,, (Mark Twain, was born in, Florida). As Open IE is significant for constructing KBs, many effective extractors have been proposed to extract triples, such as Text-Runner Yates et al. (2007), ReVerb Fader et al. (2011), and Standford Open IE Angeli et al. (2015). However, these extractors only yield relation patterns between entities, without aggregating and clustering their results. Accordingly, there are a fair amount of redundant relation patterns after extracting those relation patterns. Furthermore, the redundant patterns lead to some redundant relations in KBs.

Recently, some efforts are devoted to Open Relation Extraction (Open RE) Lin and Pantel (2001); Yao et al. (2011); Marcheggiani and Titov (2016); ElSahar et al. (2017), aiming to cluster relation patterns into several relation types instead of redundant relation patterns. Whenas, these Open RE methods adopt distantly supervised labels as golden relation types, suffering from both false positive and false negative problems on the one hand. On the other hand, these methods still rely on the conventional similarity metrics mentioned above.

In this section, we will show that our defined similarity quantification could help Open IE by identifying redundant relations. To be specific, we set a toy experiment to remove redundant relations in KBs for a preliminary comparison (§ 6.1). Then, we evaluate our model and baselines on the real-world dataset extracted by Open IE methods (§ 6.2). Considering the existing evaluation metric for Open IE and Open RE rely on either labor-intensive annotations or distantly supervised annotations, we propose a metric approximating recall and precision evaluation based on operable human annotations for balancing both efficiency and accuracy.

6.1 Toy Experiment

In this subsection, we propose a toy environment to verify our similarity-based method. Specifically, we construct a dataset from Wikidata121212The construction procedure is shown in § 4.1. and implement Chinese restaurant process131313Chinese restaurant process is shown in Appendix B. to split every relation in the dataset into several sub-relations. Then, we filter out those sub-relations appearing less than 50 times to eventually get 1165 relations. All these split relations are regarded as different ones during training, and then different relation similarity metrics are adopted to merge those sub-relations into one relation. As Figure 2 shown that the matrices-based approach is less effective than other approaches, we leave this approach out of this experiment. The results are shown in Table 4.

6.2 Real World Experiment

In this subsection, we evaluate various relation similarity metrics on the real-world Open IE patterns. The dataset are constructed by ReVerb. Different patterns will be regarded as different relations during training, and we also adopt various relation similarity metrics to merge similar relation patterns. Because it is nearly impossible to annotate all pattern pairs for their merging or not, meanwhile it is also inappropriate to take distantly supervised annotations as golden results. Hence, we propose a novel metric approximating recall and precision evaluation based on minimal human annotations for evaluation in this experiment.

Approximating Recall and Precision

Recall

Recall is defined as the yielding fraction of true positive instances over the total amount of real positive141414Often called relevant in information retrieval field. instances. However, we do not have annotations about which pairs of relations are synonymous. Crowdsourcing is a method to obtain a large number of high-quality annotations. Nevertheless, applying crowdsourcing is not trivial in our settings, because it is intractable to enumerate all synonymous pairs in the large space of relation (pattern) pairs $\mathcal{O}(|\mathcal{R}|^{2})$ in Open IE. A promising method is to use rejection sampling by uniform sampling from the whole space, and only keep the synonymous ones judged by crowdworkers. However, this is not practical either, as the synonymous pairs are sparse in the whole space, resulting in low efficiency. Fortunately, we could use normalized importance sampling as an alternative to get an unbiased estimation of recall.

Theorem 1**.**

151515See proof in Appendix A

Suppose every sample $x\in X$ has a label $f(x)\in\{0,1\}$, and the model to be evaluated also gives its prediction $\hat{f}(x)\in\{0,1\}$. The recall can be written as

[TABLE]

where $U$ is the uniform distribution over all samples with $f(x)=1$. If we have a proposal distribution $q(x)$ satisfying $\forall x,f(x)=1\wedge\hat{f}(x)=1\Rightarrow q(x)\neq 0$, we get an unbiased estimation of recall:

[TABLE]

where $\hat{w}_{i}$ is a normalized version of $w_{i}=\frac{\mathbb{I}[f(x_{i})=1]}{\tilde{q}(x_{i})}$, where $\tilde{q}$ is the unnormalized version of q, and $\{x_{i}\}_{i=1}^{n}$ are i.i.d. drawn from $q(x)$.

Precision

Similar to equation 9, we can write the expectation form of precision:

[TABLE]

where $U^{\prime}$ is the uniform distribution over all samples with $\hat{f}(x)=1$. As these samples could be found out by performing models on it. We can simply approximate precision by Monte Carlo Sampling:

[TABLE]

where $\{x_{i}\}_{i=1}^{n}\stackrel{{\scriptstyle i.i.d.}}{{\sim}}U^{\prime}$.

In our setting, $x=(r_{1},r_{2})\in\mathcal{R}\times\mathcal{R}$, $f(x)=1$ means $r_{1}$ and $r_{2}$ are the same relations, $\hat{f}(x)=1$ means $S(r_{1},r_{2})$ is larger than a threshold $\lambda$.

Results

The results on the ReVerb Extractions dataset that we constructed are described in Figure 3. To approximate recall, we use the similarity scores as the proposal distribution $\tilde{q}$. 500 relation pairs are then drawn from $\tilde{q}$. To approximate precision, we set thresholds at equal intervals. At each threshold, we uniformly sample 50 to 100 relation pairs whose similarity score given by the model is larger than the threshold. We ask 15 undergraduates to judge whether two relations in a relation pair have the same meaning. A relation pair is viewed valid only if 8 of the annotators annotate it as valid. We use the annotations to approximate recall and precision with equation 10 and equation 12. Apart from the confidential interval of precision shown in the figure, the largest $95\%$ confidential interval among thresholds for recall is $0.04$161616The figure is shown in Figure 6. From the result, we could see that our model performs much better than other models’ similarity by a very large margin.

7 Error Analysis for Relational Classification

In this section, we consider two kinds of relational classification tasks: (1) relation prediction and (2) relation extraction. Relation prediction aims at predicting the relationship between entities with a given set of triples as training data; while relation extraction aims at extracting the relationship between two entities in a sentence.

7.1 Relation Prediction

We hope to design a simple and clear experiment setup to conduct error analysis for relational prediction. Therefore, we consider a typical method TransE Bordes et al. (2013) as the subject as well as FB15K Bordes et al. (2013) as the dataset. TransE embeds entities and relations as vectors, and train these embeddings by minimizing

[TABLE]

where $\mathcal{D}$ is the set of training triples, $d(\cdot,\cdot)$ is the distance function, $(h^{\prime},r^{\prime},t^{\prime})$171717Note that only head and tail entities are changed in the original TransE when doing link prediction. But changing $r^{\prime}$ results in better performance when doing relation prediction. is a negative sample with one element different from $(h,r,t)$ uniformly sampled from $\mathcal{E}\times\mathcal{R}\times\mathcal{E}$, and $\gamma$ is the margin.

During testing, for each entity pair $(h,t)$, TransE rank relations according to $d(\bm{h}+\bm{r},\bm{t})$. For each $(h,r,t)$ in the test set, we call the relations with higher rank scores than $r$ distracting relations. We then compare the similarity between the golden relation and distracting relations. Note that some entity pairs could correspond to more than one relations, in which case we just do not see them as distracting relations.

7.2 Relation Extraction

For relation extraction, we consider the supervised relation extraction setting and TACRED dataset Zhang et al. (2017). As for the subject model, we use the best model on TACRED dataset — position-aware neural sequence model. This method first passes the sentence into an LSTM and then calculate an attention sum of the hidden states in the LSTM by taking positional features into account. This simple and effective method achieves the best in TACRED dataset.

7.3 Results

Figure 5 shows the distribution of similarity ranks of distracting relations of the above mentioned models’ outputs on both relation prediction and relation extraction tasks. From Figures 4(a) and 4(b), we could observe the most distracting relations are the most similar ones, which corroborate our hypothesis that even the best models on these tasks still make mistakes among the most similar relations. This result also highlights the importance of a heuristic method for guiding models to pay more attention to the boundary between similar relations. We also try to do the negative sampling with relation type constraints, but we see no improvement compared with uniform sampling. The details of negative sampling with relation type constraints are presented in Appendix E.

8 Similarity and Negative Sampling

Based on the observation presented in § 7.3, we find out that similar relations are often confusing for relation prediction models. Therefore, corrupted triples with similar relations can be used as high-quality negative samples.

For a given valid triple $(h,r,t)$, we corrupt the triple by substituting $r$ with $r^{\prime}$ with the probability,

[TABLE]

where $\alpha$ is the temperature of the exponential function, the bigger the $\alpha$ is, the flatter the probability distribution is. When the temperature approaches infinite, the sampling process reduces to uniform sampling.

In training, we set the initial temperature to a high level and gradually reduce the temperature. Intuitively, it enables the model to distinguish among those obviously different relations in the early stage and gives more and more confusing negative triples as the training processes to help the model distinguish the similar relations. This can be also viewed as a process of curriculum learningBengio et al. (2009), the data fed to the model gradually changes from simple negative triples to hard ones.

We perform relation prediction task on FB15K with TransE. Following Bordes et al. (2013), we use the "Filtered" setting protocol, i.e., filtering out the corrupted triples that appear in the dataset. Our sampling method is shown to improve the model’s performance, especially on Hit@1 (Figure 5). Training details are described in Appendix C.

9 Similarity and Softmax-Margin Loss

Similar to § 8, we find out that relation extraction models often make wrong preditions on similar relations. In this section, we use similarity as an adaptive margin in softmax-margin loss to improve the performance of relation extraction models.

As shown in Gimpel and Smith (2010), Softmax-Margin Loss can be expressed as

[TABLE]

where $\mathcal{R}(x)$ denotes a structured output space for $x$, and $\langle x^{(i)},r^{(i)}\rangle$ is $i^{th}$ example in training data.

We can easily incorporate similarity into cost function $\text{cost}(r^{(i)},r)$. In this task, we define the cost function as $\alpha S(r^{(i)},r)$, where $\alpha$ is a hyperparameter.

Intuitively, we give a larger margin between similar relations, forcing the model to distinguish among them, and thus making the model perform better. We apply our method to Position-aware Attention LSTM (PA-LSTM)Zhang et al. (2017), and Table 5 shows our method improves the performance of PA-LSTM. Training details are described in Appendix C.

10 Related Works

As many early works devoted to psychology and linguistics, especially those works exploring semantic similarity Miller and Charles (1991); Resnik (1999), researchers have empirically found there are various different categorizations of semantic relations among words and contexts. For promoting research on these different semantic relations, Bejar et al. (1991) explicitly defining these relations and Miller (1995) further systematically organize rich semantic relations between words via a database. For identifying correlation and distinction between different semantic relations so as to support learning semantic similarity, various methods have attempted to measure relational similarity Turney (2005, 2006); Zhila et al. (2013); Pedersen (2012); Rink and Harabagiu (2012); Mikolov et al. (2013b, a).

With the ongoing development of information extraction and effective construction of KBs Suchanek et al. (2007); Bollacker et al. (2008); Bizer et al. (2009), relations are further defined as various types of latent connections between objects more than semantic relations. These general relations play a core role in expressing relational facts in the real world. Hence, there are accordingly various methods proposed for discovering more relations and their facts, including open information extraction Brin (1998); Agichtein and Gravano (2000); Ravichandran and Hovy (2002); Banko et al. (2007); Zhu et al. (2009); Etzioni et al. (2011); Saha et al. (2017) and relation extraction Riedel et al. (2013); Liu et al. (2013); Zeng et al. (2014); Santos et al. (2015); Zeng et al. (2015); Lin et al. (2016), and relation prediction Bordes et al. (2013); Wang et al. (2014); Lin et al. (2015b, a); Xie et al. (2016).

For both semantic relations and general relations, identifying them is a crucial problem, requiring systems to provide a fine-grained relation similarity metric. However, the existing methods suffer from sparse data, which makes it difficult to achieve an effective and stable similarity metric. Motivated by this, we propose to measure relation similarity by leveraging their fact distribution so that we can identify nuances between similar relations, and merge those distant surface forms of the same relations, benefitting the tasks mentioned above.

11 Conclusion and Future Work

In this paper, we introduce an effective method to quantify the relation similarity and provide analysis and a survey of applications. We note that there are a wide range of future directions: (1) human prior knowledge could be incorporated into the similarity quantification; (2) similarity between relations could also be considered in multi-modal settings, e.g., extracting relations from images, videos, or even from audios; (3) by analyzing the distributions corresponding to different relations, one can also find some “meta-relations” between relations, such as hypernymy and hyponymy.

Acknowledgements

This work is supported by the National Natural Science Foundation of China (NSFC No. 61572273, 61532010), the National Key Research and Development Program of China (No. 2018YFB1004503). Chen and Zhu is supported by Tsinghua University Initiative Scientific Research Program, and Chen is also supported by DCST Student Academic Training Program. Han is also supported by 2018 Tencent Rhino-Bird Elite Training Program.

Appendix A Proofs to theorems in the paper

Proof.

[TABLE]

If we have a proposal distribution $q(x)$ satisfying $\forall x,f(x)=1\wedge\hat{f}(x)=1\Rightarrow q(x)\neq 0$, then equation 16 can be further written as

[TABLE]

Sometimes, it’s hard for us to compute normalized probability $q$. To tackle this problem, consider self-normalized importance sampling as an unbiased estimation Owen (2013),

[TABLE]

where $\hat{w}_{i}$ is the normalized version of $w$. ∎

Appendix B Chinese Restaurant Process

Specifically, for a relation $r$ with currently $m$ sub-relations, we turn it to a new sub-relation with probability

[TABLE]

or to the $k^{th}$ existing sub-relation with probability

[TABLE]

where $n_{k}$ is the size of $k^{th}$ existing sub-relation, $n$ is the sum of the number of all sub-relationships of $r$, and $\alpha$ is a hyperparameter, in which case we use $\alpha=1$.

Appendix C Training Details

In Wikidata and ReVerb Extractions dataset, we manually split a validation set, assuring every entity and relation appears in validation set also appears in training set. While minimizing loss on the training set, we observe the loss on the validation set and stop training as validation loss stops to decrease. Before training our model on any dataset, we use the entity embeddings and relation embeddings produced by TransE on the dataset as the pretrained embeddings for our model.

C.1 Training Details on Negative Sampling

The sampling is launched with an initial temperature of 8192. The temperature drops to half every 200 epochs and remains stable once it hits 16. Optimization is performed using SGD, with a learning rate of 1e-3.

C.2 Training Details on Softmax-Margin Loss

The sampling is launching with an initial temperature of 64. The temperature drops by 20% per epoch, and remains stable once it hits 16. The alpha we use is 9. Optimization is performed using SGD, with a learning rate of 1.

Appendix D Recall Standard Deviation

As is shown in Figure 6, the max recall standard deviation for our model is 0.4, and 0.11 for TransE.

Appendix E Negative Samplilng with Relation Type Constraints

In FB15K, if two relations have same prefix, we regard them as belonging to a same type, e.g., both /film/film/starring./film/performance/actor and /film/actor/film./film/performance/film have prefix film, they belong to same type. Similar to what is mentioned in § 8, we expect the model first to learn to distinguish among obviously different relations, and gradually learn to distinguish similar relations. Therefore, we conduct negative sampling with relation type constraints in two ways.

E.1 Add Up Two Uniform Distribution

For each triple $(h,r,t)$, we have two uniform distribution $U_{all}$ and $U_{type}$. $U_{all}$ is the uniform distribution over all the relations except for those appear with $(h,t)$ in the knowledge base, and $U_{type}$ is the uniform distribution over the relations of the same type as $r$. When corrupting the triple, we sample $r^{\prime}$ from the distribution:

[TABLE]

where $\alpha$ is a hyperparameter. We set $\alpha$ to 1 at the beginning of training, and every $k$ epochs, $\alpha$ will be multiplied by decrease rate $\gamma$. We do grid search for $k\in\{50,70,100\}$ and $\gamma\in\{0.9,0.95,0.98\}$, but no improvement is observed.

E.2 Add Weight

We speculate that the unsatisfactory result produced by adding up two uniform distribution is because that for those types with few relations in it, a small change of $\alpha$ will result in a significant change in $U$. Therefore, when sampling a negative $r^{\prime}$, we add weights to relations that are of the same type as $r$ instead. Concretely, we substitute $r$ with $r^{\prime}$ with probability $p$, which can be calculated as:

[TABLE]

where $\mathcal{T}(r)$ denotes all the relations that are the same type as $r$, $\epsilon$ is a hyperparameter and $N$ is a normalizing constant. We set $\epsilon$ to 0 at the beginning of training, and every $k$ epochs, $\epsilon$ will increase by $\gamma$. We do grid search for $k\in\{50,70,100\}$ and $\gamma\in{0.5,1}$, still no improvement is observed.

Appendix F Wikidata annotation guidance

We show the guidance provided for the annotators here.

• •

A pair of relations should be marked as 4 points if the two relations are only two different expressions for a certain meaning.

Example: (study at, be educated at)

• •

A pair of relations should be marked as 3 points if the two relations are describing a same topic, and the entities that the two relations connect are of same type respectively.

Example: (be the director of, be the screenwriter of), both relations relate to movie, and the types of the entities they connect are both (person, movie).

• •

A pair of relations should be marked as 2 points if the two relations are describing a same topic, but the entities that the two relations connect are of different type respectively.

Example: (be headquartered in, be founded in), both relations relate to organization, but the types of the entities they connect are different, i.e., (company, location) and (company, time)

• •

A pair of relations should be marked as 1 points if the two relations do not meet the conditions above but still have semantic relation.

Example: (be the developer of, be the employer of)

• •

A pair of relations should be marked as 0 points if the two relations do not have any connection.

Example: (be a railway station locates in, be published in)