CESI: Canonicalizing Open Knowledge Bases using Embeddings and Side Information

TL;DR

CESI is a novel method that uses learned embeddings and side information to canonicalize open knowledge bases, reducing redundancy and ambiguity in extracted facts.

Contribution

CESI introduces a new embedding-based approach that incorporates side information for canonicalization, overcoming manual feature engineering limitations.

Findings

CESI outperforms existing clustering methods on real-world datasets.

Embedding-based canonicalization improves accuracy and reduces ambiguity.

CESI effectively leverages side information to enhance knowledge base quality.

Abstract

Open Information Extraction (OpenIE) methods extract (noun phrase, relation phrase, noun phrase) triples from text, resulting in the construction of large Open Knowledge Bases (Open KBs). The noun phrases (NPs) and relation phrases in such Open KBs are not canonicalized, leading to the storage of redundant and ambiguous facts. Recent research has posed canonicalization of Open KBs as clustering over manuallydefined feature spaces. Manual feature engineering is expensive and often sub-optimal. In order to overcome this challenge, we propose Canonicalization using Embeddings and Side Information (CESI) - a novel approach which performs canonicalization over learned embeddings of Open KBs. CESI extends recent advances in KB embedding by incorporating relevant NP and relation phrase side information in a principled manner. Through extensive experiments on multiple real-world datasets, we demonstrate CESI's effectiveness.

Figures

Tables

Table 1. Table 1. Details of datasets used. ReVerb45K is the new dataset we propose in this paper. Please see Section  7.1 for details.

Datasets	# Gold	#NPs	#Relations	#Triples
	Entities
Base	150	290	3K	9K
Ambiguous	446	717	11K	37K
ReVerb45K	7.5K	15.5K	22K	45K

Table 2. Table 2. NP Canonicalization Results. CESI outperforms all other methods across datasets (Best in 7 out of 9 cases. Section  8.1.1 )

Method	Base Dataset			Ambiguous Dataset			ReVerb45K
	Macro	Micro	Pair.	Macro	Micro	Pair.	Macro	Micro	Pair.	Row Average
Morph Norm	58.3	88.3	83.5	49.1	57.2	70.9	1.4	77.7	75.1	62.3
PPDB	42.4	46.9	32.2	37.3	60.2	69.3	46.0	45.4	64.2	49.3
EntLinker	54.9	65.1	75.2	49.7	83.2	68.8	62.8	81.8	80.4	69.1
Galárraga-StrSim	88.2	96.5	97.7	66.6	85.3	82.2	69.9	51.7	0.5	70.9
Galárraga-IDF	94.8	97.9	98.3	67.9	82.9	79.3	71.6	50.8	0.5	71.5
Galárraga-Attr	76.1	51.4	18.1	82.9	27.7	8.4	75.1	20.1	0.2	40.0
GloVe	95.7	97.2	91.1	65.9	89.9	90.1	56.5	82.9	75.3	82.7
HolE (Random)	69.5	91.3	86.6	53.3	85.0	75.1	5.4	74.6	50.9	65.7
HolE (GloVe)	75.2	93.6	89.3	53.9	85.4	76.7	33.5	75.8	51.0	70.4
CESI	98.2	99.8	99.9	66.2	92.4	91.9	62.7	84.4	81.9	86.3

Table 3. Table 3. Relation canonicalization results. Compared to AMIE, CESI canonicalizes more number of relation phrases at higher precision. Please see Section  8.1.2 for details.

	Macro	Micro	Pairwise	Induced
	Precision	Precision	Precision	Relation
				Clusters
Base Dataset
AMIE	42.8	63.6	43.0	7
CESI	88.0	93.1	88.1	210
Ambiguous Dataset
AMIE	55.8	64.6	23.4	46
CESI	76.0	91.9	80.9	952
ReVerb45K
AMIE	69.3	84.2	66.2	51
CESI	77.3	87.8	72.6	2116

Table 4. Table 4. CESI’s performance in canonicalizing unlinked NPs, with and without Entity Linking (EL) side information, in the ReVerb45K dataset. We observe that CESI does not overfit to EL side information, and thereby helps prevent performance degradation in unlinked NP canonicalization (in fact it even helps a little). Please see Section  8.3 for details.

	Macro F1	Micro F1	Pairwise F1
CESI	81.7	87.6	81.5
CESI w/o EL	81.3	87.3	80.7

Equations

$$scor{e}_{idf}(n,n^{\prime })=\frac{{\sum }_{x\in w(n)\cap w(n^{\prime })}\log (1+f(x){)}^{-1}}{{\sum }_{x\in w(n)\cup w(n^{\prime })}\log (1+f(x){)}^{-1}}$$

$$\begin{array}{rl}\underset{\Theta }{\min }& {\lambda }_{str}\sum _{i\in D_{+}}\sum _{j\in D_{-}}\text{max}(0,\gamma +\sigma ({\eta }_j)-\sigma ({\eta }_i))\\ & +\sum _{\theta \in {\mathcal{C}}_{\text{ent}}}\frac{{\lambda }_{\text{ent},\theta }}{|{\mathcal{Z}}_{\text{ent},\theta }|}\sum _{v,v^{{}^{\prime }}\in {\mathcal{Z}}_{\text{ent},\theta }}\Vert e_v-e_{v^{{}^{\prime }}}{\Vert }^2\\ & +\sum _{\varphi \in {\mathcal{C}}_{\text{rel}}}\frac{{\lambda }_{\text{rel},\varphi }}{|{\mathcal{Z}}_{\text{rel},\varphi }|}\sum _{u,u^{{}^{\prime }}\in {\mathcal{Z}}_{\text{rel},\varphi }}\Vert r_u-r_{u^{{}^{\prime }}}{\Vert }^2\\ & +{\lambda }_{\text{reg}}\left(\sum _{v\in V}\Vert e_v{\Vert }^2+\sum _{r\in R}\Vert e_r{\Vert }^2\right).\end{array}$$

$$[a\star b{]}_k=\sum _{i=0}^{d-1}{a}_i{b}_{(k+i)\text{ mod}d}.$$

$$P_{\mathrm{macro}}(C,E)$$

$$R_{\mathrm{macro}}(C,E)$$

$$P_{\mathrm{micro}}(C,E)$$

$$R_{\mathrm{micro}}(C,E)$$

$$P_{\mathrm{pair}}(C,E)$$

$$R_{\mathrm{pair}}(C,E)$$

$$f_{\text{attr}}(n,n^{\prime })=\frac{|A\cap A^{\prime }|}{|A\cup A^{\prime }|}$$

Full text

Canonicalization; Knowledge Graphs; Knowledge Graph Embeddings; Open Knowledge Bases

††journalyear: 2018††copyright: iw3c2w3††conference: The 2018 Web Conference; April 23–27, 2018; Lyon, France††booktitle: WWW 2018: The 2018 Web Conference, April 23–27, 2018, Lyon, France††doi: 10.1145/3178876.3186030††isbn: 978-1-4503-5639-8/18/04††ccs: Computing methodologies Knowledge representation and reasoning††ccs: Computing methodologies Information extraction

1. Introduction

Recent research has resulted in the development of several large Ontological Knowledge Bases (KBs), examples include DBpedia (Auer et al., 2007), YAGO (Suchanek et al., 2007), and Freebase (Bollacker et al., 2008). These KBs are called ontological as the knowledge captured by them conform to a fixed ontology, i.e., pre-specified Categories (e.g., person, city) and Relations (e.g., mayorOfCity(Person, City)). Construction of such ontological KBs require significant human supervision. Moreover, due to the need for pre-specification of the ontology, such KB construction methods can’t be quickly adapted to new domains and corpora. While other ontological KB construction approaches such as NELL (Mitchell et al., 2015) learn from limited human supervision, they still suffers from the quick adaptation bottleneck.

In contrast, Open Information Extraction (OpenIE) methods need neither supervision nor any pre-specified ontology. Given unstructured text documents, OpenIE methods readily extract triples of the form (noun phrase, relation phrase, noun phrase) from them, resulting in the development of large Open Knowledge Bases (Open KBs). Examples of Open KBs include TextRunner (Banko et al., 2007), ReVerb (Fader et al., 2011), and OLLIE (Christensen et al., 2011; Saha et al., 2017; Mausam, 2016). While this makes OpenIE methods highly adaptable, they suffer from the following shortcoming: unlike Ontological KBs, the Noun Phrases (NPs) and relation phrases in Open KBs are not canonicalized. This results in storage of redundant and ambiguous facts.

Let us explain the need for canonicalization through a concrete example. Please consider the two sentences below.

Barack Obama was the president of US.

Obama was born in Honolulu.

Given the two sentences above, an OpenIE method may extract the two triples below and store them in an Open KB.

(Barack Obama, was president of, US)

(Obama, born in, Honolulu)

Unfortunately, neither such OpenIE methods nor the associated Open KBs have any knowledge that both Barack Obama and Obama refer to the same person. This can be a significant problem as Open KBs will not return all the facts associated with Barack Obama on querying for it. Such KBs will also contain redundant facts, which is undesirable. Thus, there is an urgent need to canonicalize noun phrases (NPs) and relations in Open KBs.

In spite of its importance, canonicalization of Open KBs is a relatively unexplored problem. In (Galárraga et al., 2014), canonicalization of Open KBs is posed as a clustering problem over manually defined feature representations. Given the costs and sub-optimality involved with manual feature engineering, and inspired by recent advances in knowledge base embedding (Bordes et al., 2013; Nickel et al., 2016), we pose canonicalization of Open KBs as a clustering over automatically learned embeddings. We make the following contributions in this paper.

• •

We propose Canonicalization using Embeddings and Side Information (CESI), a novel method for canonicalizing Open KBs using learned embeddings. To the best of our knowledge, this is the first approach to use learned embeddings and side information for canonicalizing an Open KB.

• •

CESI models the problem of noun phrase (NP) and relation phrase canonicalization jointly using relevant side information in a principled manner. This is unlike prior approaches where NP and relation phrase canonicalization were performed sequentially.

• •

We build and experiment with ReVerb45K, a new dataset for Open KB canonicalization. ReVerb45K consists of 20x more NPs than the previous biggest dataset for this task. Through extensive experiments on this and other real-world datasets, we demonstrate CESI’s effectiveness (Section 7).

CESI’s source code and datasets used in the paper are available at https://github.com/malllabiisc/cesi.

2. Related Work

Entity Linking: One traditional approach to canonicalizing noun phrases is to map them to an existing KB such as Wikipedia or Freebase. This problem is known as Entity Linking (EL) or Named Entity Disambiguation (NED). Most approaches generate a list of candidate entities for each NP and re-rank them using machine learning techniques. Entity linking has been an active area of research in the NLP community (Trani et al., 2014; Lin et al., 2012; Ratinov et al., 2011). A major problem with these kind of approaches is that many NPs may refer to new and emerging entities which may not exist in KBs. One approach to resolve these noun phrases is to map them to NIL or an OOKB (Out of Knowledge Base) entity, but the problem still remains as to how to cluster these NIL mentions. Although entity linking is not the best approach to NP canonicalization, we still leverage signals from entity linking systems for improved canonicalization in CESI.

Canonicalization in Ontological KBs: Concept Resolver (Krishnamurthy and Mitchell, 2011) is used for clustering NP mentions in NELL (Mitchell et al., 2015). It makes “one sense per category” assumption which states that a noun phrase can refer to at most one concept in each category of NELL’s ontology. For example, the noun phrase “Apple” can either refer to a company or a fruit, but it can refer to only one company and only one fruit. Another related problem to NP canonicalization is Knowledge Graph Identification (Pujara et al., 2013), where given a noisy extraction graph, the task is to produce a consistent Knowledge Graph (KG) by performing entity resolution, entity classification and link prediction jointly. Pujara et al. (Pujara et al., 2013) incorporate information from multiple extraction sources and use ontological information to infer the most probable knowledge graph using probabilistic soft logic (PSL) (Bröcheler et al., 2010). However, both of these approaches require additional information in the form of an ontology of relations, which is not available in the Open KB setting.

Relation Taxonomy Induction: SICTF (Nimishakavi et al., 2016) tries to learn relation schemas for different OpenIE relations. It is built up on RESCAL (Nickel et al., 2011), and uses tensor factorization methods to cluster noun phrases into categories (such as “person”, “disease”, etc.). We, however, are interested in clustering noun phrases into entities.

There has been relatively less work on the task of relation phrase canonicalization. Some of the early works include DIRT (Lin and Pantel, 2001), which proposes an unsupervised method for discovering inference rules of the form “*X is the author of Y * $\approx$ X wrote Y” using paths in dependency trees; and the PATTY system (Nakashole et al., 2012), which tries to learn subsumption rules among relations (such as son-of $\subset$ child-of) using techniques based on frequent itemset mining. These approaches are more focused on finding a taxonomy of relation phrases, while we are looking at finding equivalence between relation phrases.

Knowledge Base Embedding: KB embedding techniques such as TransE (Bordes et al., 2013), HolE (Nickel et al., 2016) try to learn vector space embeddings for entities and relations present in a KB. TransE makes the assumption that for any $\langle$subject, relation, object$\rangle$ triple, the relation vector is a translation from the subject vector to the object vector. HolE, on the other hand, uses non-linear operators to model a triple. These embedding methods have been successfully applied for the task of link prediction in KBs. In this work, we build up on HolE while exploiting relevant side information for the task of Open KB canonicalization. We note that, even though KB embedding techniques like HolE have been applied to ontological KBs, CESI might be the first attempt to use them in the context of Open KBs.

Canonicalizing Open KBs: The RESOLVER system (Yates and Etzioni, 2009) uses string similarity based features to cluster phrases in TextRunner (Banko et al., 2007) triples. String similarity features, although being effective, fail to handle synonymous phrases which have completely different surface forms, such as Myopia and Near-sightedness.

KB-Unify (Delli Bovi et al., 2015) addresses the problem of unifying multiple Ontological and Open KBs into one KB. However, KB-Unify requires a pre-determined sense inventory which is not available in the setting CESI operates.

The most closely related work to ours is (Galárraga et al., 2014). They perform NP canonicalization by performing Hierarchical Agglomerative Clustering (HAC) (Tan et al., 2005) over manually-defined feature spaces, and subsequently perform relation phrase clustering by using the AMIE algorithm (Galárraga et al., 2013). CESI significantly outperforms this prior method (Section 7).

3. Proposed Approach: CESI

Overall architecture and dataflow of CESI is shown in Figure 1. The input to CESI is an un-canonicalized Open Knowledge Base (KB) with source information for each triple. The output is a list of canonicalized noun and relation phrases, which can be used to identify equivalent entities and relations or canonicalize the KB. CESI achieves this through its three step procedure:

1. (1)

Side Information Acquisition: The goal of this step is to gather various NP and relation phrase side information for each triple in the input by running several standard algorithms on the source text of the triples. More details can be found in Section 4. 2. (2)

Embedding NP and Relation Phrases: In this step, CESI learns specialized vector embeddings for all NPs and relation phrases in the input by making principled use of side information available from the previous step. 3. (3)

Clustering Embeddings and Canonicalization: Goal of this step is to cluster the NPs and relation phrases on the basis of their distance in the embedding space. Each cluster represents a specific entity or relation. Based on certain relevant heuristics, we assign a representative to each NP and relation phrase cluster.

Details of different steps of CESI are described next.

4. Side Information Acquisition

Noun and relation phrases in Open KBs often have relevant side information in the form of useful context in the documents from which the triples were extracted. Sometimes, such information may also be present in other related KBs. Previous Open KB canonicalization methods (Galárraga et al., 2014) ignored such available side information and performed canonicalization in isolation focusing only on the Open KB triples. CESI attempts to exploit such side information to further improve the performance on this problem. In CESI, we make use of five types of NP side information to get equivalence relations of the form $e_{1}\equiv e_{2}$ between two entities $e_{1}$ and $e_{2}$. Similarly, relation phrase side information is used to derive relation equivalence, $r_{1}\equiv r_{2}$. All equivalences are used as soft constraints in later steps of CESI (details in Section 5).

4.1. Noun Phrase side Information

In the present version of CESI, we make use of the following five types of NP side information:

1. (1)

Entity Linking: Given unstructured text, entity linking algorithms identify entity mentions and link them to Ontological KBs such as Wikipedia, Freebase etc. We make use of Stanford CoreNLP entity linker which is based on (Spitkovsky and Chang, 2012) for getting NP to Wikipedia entity linking. Roughly, in about 30% cases, we get this information for NPs. If two NPs are linked to the same Wikipedia entity, we assume them to be equivalent as per this information. For example, US and America can get linked to the same Wikipedia entity United_States. 2. (2)

PPDB Information: We make use of PPDB 2.0 (Pavlick et al., 2015), a large collection of paraphrases in English, for identifying equivalence relation among NPs. We first extracted high confidence paraphrases from the dataset while removing duplicates. Then, using union-find, we clustered all the equivalent phrases and randomly assigned a representative to each cluster. Using an index created over the obtained clusters, we find cluster representative for each NP. If two NPs have the same cluster representative then they are considered to be equivalent. NPs not present in the dataset are skipped. This information helps us identifying equivalence between NPs such as management and administration. 3. (3)

WordNet with Word-sense Disambiguation: Using word-sense disambiguation (Banerjee and Pedersen, 2002) with Wordnet (Miller, 1995), we identify possible synsets for a given NP. If two NPs share a common synset, then they are marked as similar as per this side information. For example, picture and image can get linked to the same synset visualize.v.01. 4. (4)

IDF Token Overlap: NPs sharing infrequent terms give a strong indication of them referring to the same entity. For example, it is very likely for Warren Buffett and Buffett to refer to the same person. In (Galárraga et al., 2014), IDF token overlap was found to be the most effective feature for canonicalization. We assign a score for every pair of NPs based on the standard IDF formula:

[TABLE]

Here, $w(\cdot)$ for a given NP returns the set of its terms, excluding stop words. $f(\cdot)$ returns the document frequency for a token. 5. (5)

Morph Normalization: We make use of multiple morphological normalization operations like tense removal, pluralization, capitalization and others as used in (Fader et al., 2011) for finding out equivalent NPs. We show in Section 8.2 that this information helps in improving performance.

4.2. Relation Phrase Side Information

Similar to noun phrases, we make use of PPDB and WordNet side information for relation phrase canonicalization as well. Apart from these, we use the following two additional types of side information involving relation phrases.

1. (1)

AMIE Information: AMIE algorithm (Galárraga et al., 2013) tries to learn implication rules between two relations $r$ and $r^{\prime}$ of the form $r\Rightarrow r^{\prime}$. These rules are detected based on statistical rule mining, for more details refer (Galárraga et al., 2014). It declares two relations $r$ and $r^{\prime}$ to be equivalent if both $r\Rightarrow r^{\prime}$ and $r^{\prime}\Rightarrow r$ satisfy support and confidence thresholds. AMIE accepts a semi-canonicalized KB as input, i.e., a KB where NPs are already canonicalized. Since this is not the case with Open KBs, we first canonicalized NPs morphologically and then applied AMIE over the NP-canonicalized KB. We chose morphological normalization for this step as such normalization is available for all NPs, and also because we found this side information to be quite effective in large Open KBs. 2. (2)

KBP Information: Given unstructured text, Knowledge Base Population (KBP) systems detect relations between entities and link them to relations in standard KBs. For example, “Obama was born in Honolulu” contains “was born in” relation between Obama and Honolulu, which can be linked to per:city_of_birth relation in KBs. In CESI, we use Stanford KBP (Surdeanu et al., 2012) to categorize relations. If two relations fall in the same category, then they are considered equivalent as per this information.

The given list can be further extended based on the availability of other side information. For the experiments in this paper, we have used the above mentioned NP and relation phrase side information. Some of the equivalences derived from different side information might be erroneous, therefore, instead of using them as hard constraints, we try to use them as supplementary information as described in the next section. Even though side information might be available only for a small fraction of NPs and relation phrases, the hypothesis is that it will result in better overall canonicalization. We find this to be true, as shown in Section 8.

5. Embedding NP and Relation Phrases

For learning embeddings of NPs and relation phrases in a given Open KB, CESI optimizes HolE’s (Nickel et al., 2016) objective function along with terms for penalizing violation of equivalence conditions from the NP and relation phrase side information. Since the conditions from side information might be spurious, a factor ($\lambda_{\text{ent}/\text{rel},\theta}$) is multiplied with each term, which acts as a hyper-parameter and is tuned on a held out validation set. We also keep a constant ($\lambda_{str}$) with HolE objective function, to make selective use of structural information from KB for canonicalization. We choose HolE because it is one of the best performing KB embeddings techniques for tasks like link prediction in knowledge graphs. Since KBs store only true triples, we generate negative examples using local closed world heuristic (Dong et al., 2014). To keep the rank of true triples higher than the non-existing ones, we use pairwise ranking loss function. The final objective function is described below.

[TABLE]

The objective function, consists of three main terms, along with one term for regularization. Optimization parameter, $\Theta=\{e_{v}\}_{v\in V}\cup\{r_{u}\}_{u\in R}$, is the set of all NP ($e_{v}$) and relation phrase ($r_{u}$) $d$-dimensional embeddings, where, $V$ and $R$ denote the set of all NPs and relation phrases in the input. In the first term, $D_{+},D_{-}$ specify the set of positive and negative examples and $\gamma>0$ refers to the width of the margin (Bordes et al., 2013). Further, $\sigma(\cdot)$ denotes the logistic function and for a triple $t_{i}$ $(s,p,o)$, $\eta_{i}=r_{p}^{T}(e_{s}\star e_{o})$, where $\star:R^{d}\times R^{d}\rightarrow R^{d}$ is the circular correlation operator defined as follows.

[TABLE]

The first index of $(a\star b)$ measures the similarity between $a$ and $b$, while other indices capture the interaction of features from $a$ and $b$, in a particular order. Please refer to (Nickel et al., 2016) for more details.

In the second and third terms, $\mathscr{C}_{\text{ent}}$ and $\mathscr{C}_{\text{rel}}$ are the collection of all types of NP and relation side information available from the previous step (Section 4), i.e., $\mathscr{C}_{\text{ent}}=\{\text{Entity Linking, PPDB, ..}\}$ and $\mathscr{C}_{\text{rel}}=\{\text{AMIE, KBP, ..}\}$. Further, $\lambda_{\text{ent},\theta}$ and $\lambda_{\text{rel},\phi}$ denote the constants associated with entity and relation side information. Their value is tuned using grid search on a held out validation set. The set of all equivalence conditions from a particular side information is denoted by $\mathcal{Z}_{\text{ent},\theta}$ and $\mathcal{Z}_{\text{rel},\phi}$. The rationale behind putting these terms is to allow inclusion of side information while learning embeddings, by enforcing two NPs or relations close together if they are equivalent as per the available side information. Since the side information is available for a fraction of NPs and relation phrases in the input, including these terms in the objective does not slow down the training of embeddings significantly.

The last term adds L2 regularization on the embeddings. All embeddings are initialized by averaging GloVe vectors (Pennington et al., 2014). We use mini-batch gradient descent for optimization.

6. Clustering Embeddings and Canonicalization

CESI clusters NPs and relation phrases by performing Hierarchical Agglomerative Clustering (HAC) using cosine similarity over the embeddings learned in the previous step (Section 5). HAC was preferred over other clustering methods because the number of clusters are not known beforehand. Complete linkage criterion is used for calculating the similarity between intermediate clusters as it gives smaller sized clusters, compared to single and average linkage criterion. This is more reasonable for canonicalization problem, where cluster sizes are expected to be small. The threshold value for HAC was chosen based on held out validation dataset.

The time complexity of HAC with complete linkage criterion is $O(n^{2})$ (Defays, 1977). For scaling up CESI to large knowledge graphs, one may go for modern variants of approximate Hierarchical clustering algorithms (Kobren et al., 2017) at the cost of some loss in performance.

Finally, we decide a representative for each NP and relation phrase cluster. For each cluster, we compute a mean of all elements’ embeddings weighted by the frequency of occurrence of each element in the input. NP or relation phrase which lies closest to the weighted cluster mean is chosen as the representative of the cluster.

7. Experimental Setup

7.1. Datasets

Statistics of the three datasets used in the experiments of this paper are summarized in Table 1. We present below brief summary of each dataset.

1. (1)

Base and Ambiguous Datasets: We obtained the Base and Ambiguous datasets from the authors of (Galárraga et al., 2014). Base dataset was created by collecting triples containing 150 sampled Freebase entities that appear with at least two aliases in ReVerb Open KB. The same dataset was further enriched with mentions of homonym entities to create the Ambiguous dataset. Please see (Galárraga et al., 2014) for more details. 2. (2)

ReVerb45K: This is the new Open KB canonicalization dataset we propose in this paper. ReVerb45K is a significantly extended version of the Ambiguous dataset, containing more than 20x NPs. ReVerb45K is constructed by intersecting information from the following three sources: ReVerb Open KB (Fader et al., 2011), Freebase entity linking information from (Gabrilovich et al., 2013), and Clueweb09 corpus (Callan et al., 2009). Firstly, for every triple in ReVerb, we extracted the source text from Clueweb09 corpus from which the triple was generated. In this process, we rejected triples for which we could not find any source text. Then, based on the entity linking information from (Gabrilovich et al., 2013), we linked all subjects and objects of triples to their corresponding Freebase entities. If we could not find high confidence linking information for both subject and object in a triple, then it was rejected. Further, following the dataset construction procedure adopted by (Galárraga et al., 2014), we selected triples associated with all Freebase entities with at least two aliases occurring as subject in our dataset. Through these steps, we obtained 45K high-quality triples which we used for evaluation. We call this resulting dataset ReVerb45K.

In contrast to Base and Ambiguous datasets, the number of entities, NPs and relation phrases in ReVerb45K are significantly larger. Please see Table 1 for a detailed comparison. This better mimics real-world KBs which tend to be sparse with very few edges per entity, as also observed by (Bordes et al., 2013).

For getting test and validation set for each dataset, we randomly sampled 20% Freebase entities and called all the triples associated with them as validation set and rest was used as the test set.

7.2. Evaluation Metrics

Following (Galárraga et al., 2014), we use macro-, micro- and pairwise metrics for evaluating Open KB canonicalization methods. We briefly describe below these metrics for completeness. In all cases, $C$ denotes the clusters produced by the algorithm to be evaluated, and $E$ denotes the gold standard clusters. In all cases, F1 measure is given as the harmonic mean of precision and recall.

Macro: Macro precision ($P_{\mathrm{macro}}$) is defined as the fraction of pure clusters in $C$, i.e., clusters in which all the NPs (or relations) are linked to the same gold entity (or relation). Macro recall ($R_{\mathrm{macro}}$) is calculated like macro precision but with the roles of $E$ and $C$ interchanged.

[TABLE]

Micro: Micro precision ($P_{\mathrm{micro}}$) is defined as the purity of $C$ clusters (Manning et al., 2008) based on the assumption that the most frequent gold entity (or relation) in a cluster is correct. Micro recall ($R_{\mathrm{micro}}$) is defined similarly as macro recall.

[TABLE]

Pairwise: Pairwise precision ($P_{\mathrm{pair}}$) is measured as the ratio of the number of hits in $C$ to the total possible pairs in $C$. Whereas, pairwise recall ($R_{\mathrm{pair}}$) is the ratio of number of hits in $C$ to all possible pairs in $E$. A pair of elements in a cluster in $C$ produce a hit if they both refer to the same gold entity (or relation).

[TABLE]

Let us illustrate these metrics through a concrete NP canonicalization example shown in Figure 2. In this Figure, we can see that only $c_{2}$ and $c_{3}$ clusters in C are pure because they contain mentions of only one entity, and hence, $P_{\mathrm{macro}}=\frac{2}{3}$. On the other hand, we have $e_{1}$ and $e_{3}$ as pure clusters if we interchange the roles of $E$ and $C$. So, $R_{\mathrm{macro}}=\frac{2}{3}$ in this case. For micro precision, we can see that America, New York, and California are the most frequent gold entities in $C$ clusters. Hence, $P_{\mathrm{micro}}=\frac{6}{7}$. Similarly, $R_{\mathrm{micro}}=\frac{6}{7}$ in this case. For pairwise analysis, we need to first calculate the number of hits in $C$. In $c_{1}$ we have 3 possible pairs out of which only 1, (America, USA) is a hit as they belong to same gold cluster $e_{1}$. Similarly, we have 3 hits in $c_{2}$ and 0 hits in $c_{3}$. Hence, $P_{\mathrm{pair}}=\frac{4}{6}$. To compute $R_{\mathrm{pair}}$, we need total number of pairwise decisions in $E$, which is $1+6+0$ , thus, $R_{\mathrm{pair}}=\frac{4}{7}$. All the results are summarized in Table 2.

For evaluating NP canonicalization, we use Macro, Micro and Pairwise F1 score. However, in the case of relations, where gold labels are not available, we use macro, micro and pairwise precision values based on the scores given by human judges.

7.3. Methods Compared

7.3.1. Noun Phrase Canonicalization

For NP canonicalization, CESI has been compared against the following methods:

• •

Morphological Normalization: As used in (Fader et al., 2011), this involves applying simple normalization operations like removing tense, pluralization, capitalization etc. over NPs and relation phrases.

• •

Paraphrase Database (PPDB): Using PPDB 2.0 (Pavlick et al., 2015), we clustered two NPs together if they happened to share a common paraphrase. NPs which could not be found in PPDB are put into singleton clusters.

• •

Entity Linking: Since the problem of NP canonicalization is closely related to entity linking, we compare our method against Stanford CoreNLP Entity Linker (Spitkovsky and Chang, 2012). Two NPs linked to the same entity are clustered together.

• •

Galárraga-IDF (Galárraga et al., 2014): IDF Token Overlap was the best performing method proposed in (Galárraga et al., 2014) for NP canonicalization. In this method, IDF token similarity is defined between two NPs as in Section 4.1, and HAC is used to cluster the mentions.

• •

Galárraga-StrSim (Galárraga et al., 2014): This method is similar to Galarraga-IDF, but with similarity metric being the Jaro-Winkler (Winkler, 1999) string similarity measure.

• •

Galárraga-Attr (Galárraga et al., 2014): Again, this method is similar to the Galarraga-IDF, except that Attribute Overlap is used as the similarity metric between two NPs in this case. Attribute for a NP $n$, is defined as the set of relation-NP pairs which co-occur with $n$ in the input triples. Attribute overlap similarity between two NPs, is defined as the Jaccard coefficient of the set of attributes:

[TABLE]

where, $A$ and $A^{\prime}$ denote the set of attributes associated with $n$ and $n^{\prime}$.

Since canonicalization methods using above similarity measures were found to be most effective in (Galárraga et al., 2014), even outperforming Machine Learning-based alternatives, we consider these three baselines as representatives of state-of-the-art in Open KB canonicalization.

• •

GloVe: In this scheme, each NP and relation phrase is represented by a 300 dimensional GloVe embedding (Pennington et al., 2014) trained on Wikipedia 2014 and Gigaword 5 (Parker et al., 2011) datasets with 400k vocabulary size. Word vectors were averaged together to get embeddings for multi-word phrases. These GloVE embeddings were then clustered for final canonicalization.

• •

HolE: In this method, embeddings of NPs and relation phrases in an Open KB are obtained by applying HolE (Nickel et al., 2016) over the Open KB. These embeddings are then clustered to obtain the final canonicalized groupings. Based on the initialization of embeddings, we differentiate between HolE(Random) and HolE(GloVe).

• •

CESI: This is the method proposed in this paper, please see Section 3 for more details.

Hyper-parameters: Following (Galárraga et al., 2014), we used Hierarchical Agglomerative Clustering (HAC) as the default clustering method across all methods (wherever necessary). For all methods, grid search over the hyperparameter space was performed, and results for the best performing setting are reported. This process was repeated for each dataset.

7.3.2. Relation Phrase Canonicalization

AMIE (Galárraga et al., 2013) was found to be effective for relation phrase canonicalization in (Galárraga et al., 2014). We thus consider AMIE111We use support and confidence values of 2 and 0.2 for all the experiments in this paper. as the state-of-the-art baseline for relation phrase canonicalization and compare against CESI. We note that AMIE requires NPs of the input Open KB to be already canonicalized. In all our evaluation datasets, we already have gold NP canonicalization available. We provide this gold NP canonicalization information as input to AMIE. Please note that CESI doesn’t require such pre-canonicalized NP as input, as it performs joint NP and relation phrase canonicalization. Moreover, providing gold NP canonicalization information to AMIE puts CESI at a disadvantage. We decided to pursue this choice anyways in the interest of stricter evaluation. However, in spite of starting from this disadvantageous position, CESI significantly outperforms AMIE in relation phrase canonicalization, as we will see in Section 8.1.2.

For evaluating performance of both algorithms, we randomly sampled 25 non-singleton relation clusters for each of the three datasets and gave them to five different human evaluators222Authors did not participate in this evaluation. for assigning scores to each cluster. The setting was kept blind, i.e., identity of the algorithm producing a cluster was not known to the evaluators. Based on the average of evaluation scores, precision values were calculated. Only non-singleton clusters were sampled, as singleton clusters will always give a precision of one.

8. Results

In this section, we evaluate the following questions.

• Q1.

Is CESI effective in Open KB canonicalization? (Section 8.1)

• Q2.

What is the effect of side information in CESI’s performance? (Section 8.2)

• Q3.

Does addition of entity linking side information degrade CESI’s ability to canonicalize unlinked NPs (i.e., NPs missed by the entity linker)? (Section 8.3)

Finally, in Section 8.4, we present qualitative examples and discussions.

8.1. Evaluating Effectiveness of CESI in Open KB Canonicalization

8.1.1. Noun Phrase Canonicalization

Results for NP canonicalization are summarized in Table 2. Overall, we find that CESI performs well consistently across the datasets. Morphological Normalization failed to give competitive performance in presence of homonymy. PPDB, in spite of being a vast reservoir of paraphrases, lacks information about real-world entities like people, places etc. Therefore, its performance remained weak throughout all datasets. Entity linking methods make use of contextual information from source text of each triple to link a NP to a KB entity. But their performance is limited because they are restricted by the entities in KB. String similarity also gave decent performance in most cases but since they solely rely on surface form of NPs, they are bound to fail with NPs having dissimilar mentions.

Methods such as Galárraga-IDF, Galárraga-StrSim, and Galárraga-Attr performed poorly on ReVerb45K. Although, their performance is considerably better on the other two datasets. This is because of the fact that in contrast to Base and Ambiguous datasets, ReVerb45K has considerably large number of entities and comparatively fewer triples (Table 1). Galárraga-IDF token overlap is more likely to put two NPs together if they share an uncommon token, i.e., one with high IDF value. Hence, accuracy of the method relies heavily on the quality of document frequency estimates which may be quite misleading when we have smaller number of triples. Similar is the case with Galárraga-Attr which decides similarity of NPs based on the set of shared attributes. Since, attributes for a NP is defined as a set of relation-NP pairs occurring with it across all triples, sparse data also results in poor performance for this method.

GloVe captures semantics of NPs and unlike string similarity it doesn’t rely on the surface form of NPs. Therefore, its performance has been substantial across all the datasets. HolE captures structural information from the given triples and uses it for learning embeddings. Through our experiments, we can see that solely structural information from KB is quite effective for NP canonicalization. CESI performs the best across the datasets in 7 out of the 9 settings, as it incorporates the strength of all the listed methods. The superior performance of CESI compared to HolE clearly indicates that the side information is indeed helpful for canonicalization task. Results of GloVe, HolE and CESI suggest that embeddings based method are much more effective for Open KB canonicalization.

8.1.2. Relation Phrase Canonicalization

Results for relation phrase canonicalization are presented in Table 3. For all experiments, in spite of using quite low values for minimum support and confidence, AMIE was unable to induce any reasonable number of non-singleton clusters (e.g., only 51 clusters out of the 22K relation phrases in the ReVerb45K dataset). For relation canonicalization experiments, AMIE was evaluated on gold NP canonicalized data as the algorithm requires NPs to be already canonicalized. CESI, on the other hand, was tested on all the datasets without making use of gold NP canonicalization information.

Based on the results in Table 3, it is quite evident that AMIE induces too few relation clusters to be of value in practical settings. On the other hand, CESI consistently performs well across all the datasets and induces significantly larger number of clusters.

8.2. Effect of Side Information in CESI

In this section, we evaluate the effect of various side information in CESI’s performance. For this, we evaluated the performances of various versions of CESI, each one of them obtained by ablating increasing amounts of side information from the full CESI model. Experimental results comparing these ablated versions on the ReVerb45K are presented in Figure 3. From this figure, we observe that while macro performance benefits most from different forms of side information, micro and pairwise performance also show increased performance in the presence of various side information. This validates one of the central thesis of this paper: side information, along with embeddings, can result in improved Open KB canonicalization.

8.3. Effect of Entity Linking Side Information on Unlinked NP Canonicalization

From experiments in Section 8.2, we find that Entity Linking (EL) side information (see Section 4.1) is one of the most useful side information that CESI exploits. However, such side information is not available in case of unlinked NPs, i.e., NPs which were not linked by the entity linker. So, this naturally raises the following question: does CESI overfit to the EL side information and ignore the unlinked NPs, thereby resulting in poor canonicalization of such unlinked NPs?

In order to evaluate this question, we compared CESI’s performance on unlinked NPs in the ReVerb45K dataset, with and without EL side information. We note that triples involving unlinked NPs constitute about 25% of the entire dataset. Results are presented in Table 4. From this table, we observe that CESI doesn’t overfit to EL side information, and it selectively uses such information when appropriate (i.e., for linked NPs). Because of this robust nature, presence of EL side information in CESI doesn’t have an adverse effect on the unlinked NPs, in fact there is a small gain in performance.

8.4. Qualitative Evaluation

Figure 4 shows some of the NP and relation phrase clusters detected by CESI in ReVerb45K dataset. These results highlight the efficacy of algorithm in canonicalizing non-trivial NPs and relation phrases. The figure shows t-SNE (van der Maaten and Hinton, 2008) visualization of NP and relation phrase (marked in ’$<\cdots>$’) embeddings for a few examples. We can see that the learned embeddings are actually able to capture equivalence of NPs and relation phrases. The algorithm is able to correctly embed Prozac, Sarafem and Fluoxetine together (different names of the same drug), despite their having completely different surface forms.

Figure 4 also highlights the failures of CESI. For example, Toyota and Nissan have been embedded together although the two being different companies. Another case is with Pablo and Juan Pablo Angel, which refer to different entities. The latter case can be avoided by keeping track of the source domain type information of each NP for disambiguation. In this if we know that Juan Pablo Angel has come from SPORTS domain, whereas Pablo has come from a different domain then we can avoid putting them together. We tried using DMOZ (Sood, 2016) dataset, which provide mapping from URL domain to their categories, for handling such errors. But, because of poor coverage of URLs in DMOZ dataset, we couldn’t get significant improvement in canonicalization results. We leave this as a future work.

9. Conclusion

Canonicalizing Open Knowledge Bases (KBs) is an important but underexplored problem. In this paper, we proposed CESI, a novel method for canonicalizing Open KBs using learned embeddings and side information. CESI solves a joint objective to learn noun and relation phrase embeddings, while utilizing relevant side information in a principled manner. These learned embeddings are then clustered together to obtain canonicalized noun and relation phrase clusters. In this paper, we also propose ReVerb45K, a new and larger dataset for Open KB canonicalization. Through extensive experiments on this and other real-world datasets, we demonstrate CESI’s effectiveness over state-of-the-art baselines. CESI’s source code and all data used in the paper are publicly available333https://github.com/malllabiisc/cesi.

Acknowledgement

We thank the reviewers for their constructive comments. This work is supported in part by MHRD, Govt. of India, and by gifts from Google Research and Accenture. We thank Anand Mishra and other members of MALL Lab, IISc for carefully reading drafts of this paper.