Inferring Concept Hierarchies from Text Corpora via Hyperbolic Embeddings

TL;DR

This paper introduces a novel method combining hyperbolic embeddings and Hearst patterns to infer accurate and consistent concept hierarchies from large text corpora, outperforming existing approaches.

Contribution

It presents a new approach that leverages hyperbolic space for better hierarchy inference, including handling missing links and correcting errors, with improved efficiency and consistency.

Findings

Achieves state-of-the-art performance on benchmark datasets.

Effectively predicts missing is-a relationships.

Enhances hierarchical consistency of inferred taxonomies.

Abstract

We consider the task of inferring is-a relationships from large text corpora. For this purpose, we propose a new method combining hyperbolic embeddings and Hearst patterns. This approach allows us to set appropriate constraints for inferring concept hierarchies from distributional contexts while also being able to predict missing is-a relationships and to correct wrong extractions. Moreover -- and in contrast with other methods -- the hierarchical nature of hyperbolic space allows us to learn highly efficient representations and to improve the taxonomic consistency of the inferred hierarchies. Experimentally, we show that our approach achieves state-of-the-art performance on several commonly-used benchmarks.

Tables

Table 1. Table 1: Hearst patterns used in this study. Patterns are lemmatized, but listed as inflected for clarity.

Pattern

X which is a (example $|$ class $|$ kind $|$ …) of Y

X (and $|$ or) (any $|$ some) other Y

X which is called Y

X is JJS (most)? Y

X a special case of Y

X is an Y that

X is a !(member $|$ part $|$ given) Y

!(features $|$ properties) Y such as X${}_{\text{1}}$, X${}_{\text{2}}$, …

(Unlike $|$ like) (most $|$ all $|$ any $|$ other) Y, X

Y including X${}_{\text{1}}$, X${}_{\text{2}}$, …

Table 2. Table 2: Experimental results comparing distributional and pattern-based methods in all settings.

	Detection (AP)					Direction (Acc.)			Graded ($\rho$)
	Bless	Eval	Leds	Shwartz	WBless	Bless	WBless	BiBless	Hyperlex
Cosine	.12	.29	.71	.31	.53	.00	.54	.52	.14
WeedsPrec	.19	.39	.87	.43	.68	.63	.59	.45	.43
invCL	.18	.37	.89	.38	.66	.64	.60	.47	.43
SLQS	.15	.35	.60	.38	.69	.75	.67	.51	.16
p(x, y)	.49	.38	.71	.29	.74	.46	.69	.62	.62
ppmi(x, y)	.45	.36	.70	.28	.72	.46	.68	.61	.60
sp(x, y)	.66	.45	.81	.41	.91	.96	.84	.80	.51
spmi(x, y)	.76	.48	.84	.44	.96	.96	.87	.85	.53
HypeCones	.81	.50	.89	.50	.98	.94	.90	.87	.59

Table 3. Table 3: Reconstruction of Animals, Plants, and Vehicles subtrees in WordNet .

	Animals			Plants			Vehicles
	All	Missing	Transitive	All	Missing	Transitive	All	Missing	Transitive
p(x, y)	350.18	512.28	455.27	271.38	393.98	363.73	43.12	82.57	66.10
ppmi(x, y)	350.47	512.28	455.38	271.40	393.98	363.76	43.20	82.57	66.16
sp(x, y)	56.56	77.10	11.22	43.40	64.70	17.88	9.19	26.98	14.84
spmi(x, y)	58.40	102.56	12.37	40.61	71.81	14.80	9.62	17.96	3.03
HypeCones	25.33	37.60	4.37	17.00	31.53	6.36	5.12	10.28	2.74
$\Delta \%$	56.6	51.2	61.1	58.1	51.3	57.0	44.3	42.8	9.6

Equations

$$d_p(\mathbf{u},\mathbf{v})$$

$$\delta (\mathbf{u},\mathbf{v})$$

$$\mathcal{E}(u,v)=\max (0,\varphi (\mathbf{u},\mathbf{v})-\alpha (\mathbf{v}))$$

$$\hat{\Theta }=\underset{\Theta \in {\mathbb{H}}^n}{\mathrm{argmin}}\sum _{u,v\in V}\mathcal{L}(u,v)$$

$$\mathcal{L}(u,v)=\{\begin{array}{ll}\mathcal{E}(u,v)& \text{if }(u,v)\in E\\ \max (0,\gamma -\mathcal{E}(u,v))& \text{otherwise}\end{array}$$

$$⟨\mathbf{x},\mathbf{y}{⟩}_{\mathbb{L}}=-x_0{y}_0+\sum _{i=1}^n{x}_n{y}_n$$

$${\mathbb{H}}^n=\{\mathbf{x}\in {\mathbb{R}}^{n+1}:⟨\mathbf{x},\mathbf{x}{⟩}_{\mathbb{L}}=-1,x_0>0\}$$

$$d_{\ell }(\mathbf{x},\mathbf{y})={\cosh }^{-1}(-⟨\mathbf{x},\mathbf{y}{⟩}_{\mathbb{L}})$$

$$p(x_0,x_1,\dots ,x_n)=\frac{(x_1,\dots ,x_n)}{x_0+1}$$

$$p^{-1}(x_1,\dots ,x_n)=\frac{(1+\Vert x{\Vert }^2,2x_1,\dots ,2x_n)}{1-\Vert x{\Vert }^2}$$

$$\alpha (\mathbf{v})$$

$$\varphi (\mathbf{u},\mathbf{v})$$

$${\theta }_{t+1}={\exp }_{{\theta }_t}(-\eta \mathrm{grad}f({\theta }_t))$$

$$\mathrm{grad}f({\theta }_t)={\text{proj}}_{{\theta }_t}\left(g_{\ell }^{-1}\nabla f(\theta )\right)$$

$${\text{proj}}_{\mathbf{x}}(\mathbf{u})$$

$$g_{\ell }^{-1}(\mathbf{x})$$

$${\exp }_{\mathbf{x}}(\mathbf{v})=\cosh (\Vert \mathbf{v}{\Vert }_{\mathbb{L}})\mathbf{x}+\sinh (\Vert \mathbf{v}{\Vert }_{\mathbb{L}})\frac{\mathbf{v}}{\Vert \mathbf{v}{\Vert }_{\mathbb{L}}}$$

$$\text{p}(x,y)=\frac{w(x,y)}{W}$$

$$\text{ppmi}(x,y)=max\left(0,log\frac{p(x,y)}{p^{-}(x)p^{+}(y)}\right)$$

$$\text{sp}(x,y)=u_x^{\top }{\Sigma }_r{v}_y$$

$$\text{WeedsPrec}(x,y)=\frac{{\sum }_{i=1}^n{x}_i\cdot \text{mathbbm}{1}_{y_i>0}}{{\sum }_{i=1}^n{x}_i}$$

$$\text{CL}(x,y)=\frac{{\sum }_{i=1}^n\min (x_i,y_i)}{{\sum }_{i=1}^n{x}_i}$$

$$\text{invCL}(x,y)=\sqrt{\text{CL}(x,y)\cdot (1-\text{CL}(y,x)}$$

$$E_x={\text{median}}_{i=1}^k[H(c_i)]$$

$$\text{SLQS}(x,y)=1-E_x/E_y$$

Full text

1 Introduction

Concept hierarchies, i.e., systems of is-a relationships, are ubiquitous in knowledge representation and reasoning. For instance, understanding is-a relationships between concepts is of special interest in many scientific fields as it enables high-level abstractions for reasoning and provides structural insights into the concept space of a domain. A prime example is biology, in which taxonomies have a long history ranging from Linnaeus et al. (1758) up to recent efforts such as the Gene Ontology and the UniProt taxonomy (Ashburner et al., 2000; Gene Ontology Consortium, 2016; Apweiler et al., 2004). Similarly, in medicine, ontologies like MeSH and ICD-10 are used to organize medical concepts such as diseases, drugs, and treatments (Rogers, 1963; Simms, 1992). In Artificial Intelligence, concept hierarchies provide valuable information for a wide range of tasks such as automated reasoning, few-shot learning, transfer learning, textual entailment, and semantic similarity (Resnik, 1993; Lin, 1998; Berners-Lee et al., 2001; Dagan et al., 2010; Bowman et al., 2015; Zamir et al., 2018). In addition, is-a relationships are the basis of complex knowledge graphs such as DBpedia (Auer et al., 2007) and Yago (Suchanek et al., 2007; Hoffart et al., 2013) which have found important applications in text understanding and question answering

Creating and inferring concept hierarchies has, for these reasons, been a long standing task in fields such as natural language processing, the semantic web, and artificial intelligence. Early approaches such as WordNet (Miller et al., 1990; Miller and Fellbaum, 1998) and CyC (Lenat, 1995) have focused on the manual construction of high-quality ontologies. To increase scalability and coverage, the focus in recent efforts such as ProBase (Wu et al., 2012) and WebIsaDB (Seitner et al., 2016) has shifted towards automated construction.

In this work, we consider the task of inferring concept hierarchies from large text corpora in an unsupervised way. For this purpose, we combine Hearst patterns with recently introduced hyperbolic embeddings (Nickel and Kiela, 2017, 2018), what provides important advantages for this task. First, as Roller et al. (2018) showed recently, Hearst patterns provide important constraints for hypernymy extraction from distributional contexts. However, it is also well-known that Hearst patterns are prone to missing and wrong extractions, as words must co-occur in exactly the right pattern to be detected successfully. For this reason, we first extract potential is-a relationships from a corpus using Hearst patterns and build a directed weighted graph from these extractions. We then embed this Heart Graph in hyperbolic space to infer missing hypernymy relations and remove wrong extractions. By using hyperbolic space for the embedding, we can exploit the following important advantages:

Consistency

Hyperbolic entailment cones (Ganea et al., 2018) allow us to enforce transitivity of is-a-relations in the entire embedding space. This improves the taxonomic consistency of the model, as it enforces that $(x,\texttt{is-a},z)$ if $(x,\texttt{is-a},y)$ and $(y,\texttt{is-a},z)$. To improve optimization properties, we also propose a new method to compute hyperbolic entailment cones in the Lorentz model of hyperbolic space.

Efficiency

Hyperbolic space allows for very low dimensional embeddings of graphs with latent hierarchies and heavy-tailed degree distributions. To embed large Hearst graphs – which exhibit both properties (e.g., see Figure 2) – this is an important advantage. In our experiments, we will show that hyperbolic embeddings allow us to decrease the embedding dimension by over an order of magnitude while outperforming SVD-based methods.

Interpretability

In hyperbolic embeddings, similarity is captured via distance while hierarchy is captured through the norm of embeddings. In addition to semantic similarity this allows us to get additional insights from the embedding such as the generality of terms.

Figure 1 shows an example of a two-dimensional embedding of the Hearst graph that we use in our experiments. Although we will use higher dimensionalities for our final embedding, the visualization serves as a good illustration of the hierarchical structure that is obtained through the embedding.

2 Related Work

Hypernym detection

Detecting is-a-relations from text is a long-standing task in natural language processing. A popular approach is to exploit high-precision lexico-syntactic patterns as first proposed by Hearst (1992). These patterns may be predefined or learned automatically (Snow et al., 2005; Shwartz et al., 2016). However, it is well known that such pattern-based methods suffer significantly from missing extractions as terms must occur in exactly the right configuration to be detected Shwartz et al. (2016); Roller et al. (2018). Recent works improve coverage by leveraging search engines (Kozareva and Hovy, 2010) or by exploiting web-scale corpora (Seitner et al., 2016); but also come with significant precision trade-offs.

To overcome the sparse extractions of pattern-based methods, focus has recently shifted to distributional approaches which provide rich representations of lexical meaning. These methods alleviate the sparsity issue but also require specialized similarity measures to distinguish different lexical relationships. To date, most measures are inspired by the Distributional Inclusion Hypothesis (DIH; Geffet and Dagan 2005) which hypothesizes that for a subsumption relation (cat, is-a, mammal) the subordinate term (cat) should appear in a subset of the contexts in which the superior term (mammal) occurs. Unsupervised methods for hypernymy detection based on distributional approaches include WeedsPrec (Weeds et al., 2004), invCL (Lenci and Benotto, 2012), SLQS Santus et al. (2014), and DIVE Chang et al. (2018). Distributional representations that are based on positional or dependecy-based contexts may also capture crude Hearst-pattern-like features (Levy et al., 2015; Roller and Erk, 2016). Shwartz et al. (2017) showed that such contexts plays an important role for the success of distributional methods.

Recently, Roller et al. (2018) performed a systematic study of unsupervised distributional and pattern-based approaches. Their results showed that pattern-based methods are able to outperform DIH-based methods on several challenging hypernymy benchmarks. Key aspects to good performance where the extraction of patterns from large text corpora and using embedding methods to overcome the sparsity issue. Our work builds on these findings by replacing their embeddings with ones with a naturally hierarchical geometry.

Taxonomy induction

Although detecting hypernymy relationships is an important and difficult task, these systems alone do not produce rich taxonomic graph structures Camacho-Collados (2017), and complete taxonomy induction may be seen as a parallel and complementary task.

Many works in this area consider a taxonomic graph as the starting point, and consider a variety of methods for growing or discovering areas of the graph. For example, Snow et al. (2006) train a classifier to predict the likelihood of an edge in WordNet, and suggest new undiscovered edges, while Kozareva and Hovy (2010) propose an algorithm which repeatedly crawls for new edges using a web search engine and an initial seed taxonomy. Cimiano et al. (2005) considered learning ontologies using Formal Concept Analysis. Similar works consider noisy graphs discovered from Hearst patterns, and provide algorithms for pruning edges until a strict hierarchy remains (Velardi et al., 2005; Kozareva and Hovy, 2010; Velardi et al., 2013). Maedche and Staab (2001) proposed a method to learn ontologies in a Semantic Web context.

Embeddings

Recently, works have proposed a variety of graph embedding techniques for representing and recovering hierarchical structure. Order-embeddings Vendrov et al. (2016) represent text and images with embeddings, where the ordering over individual dimensions form a partially ordered set. Hyperbolic embeddings treat words as points in non-Euclidean geometries, and may be viewed as a continuous generalization of tree structures Nickel and Kiela (2017, 2018). Extensions have considered how distributional co-occurrences may be used to augment order embeddings Li et al. (2018) and Hyperbolic embeddings Dhingra et al. (2018). Other recent works have focused on the often complex overlapping structure of word classes, and induced hierarchies using box-lattice structures Vilnis et al. (2018) and Gaussian word embeddings Athiwaratkun and Wilson (2018). Compared to many of the purely graph-based works, these methods generally require extensive supervision of hierarchical structure, and cannot learn taxonomies using only unstructured noisy data. Recently, Tifrea et al. (2018) proposed an extension of GloVe (Pennington et al., 2014) to hyperbolic space. Our experimental results in Section 4, which show substantial gains over the results reported by Tifrea et al. (2018) for hypernymy prediction, underline the importance of selecting the right distributional context for this task.

3 Methods

In the following, we discuss our method for unsupervised learning of concept hierarchies. We first discuss the extraction and construction of the Hearst graph, followed by a description of the Hyperbolic Embeddings.

3.1 Hearst Graph

The main idea introduced by Hearst (1992) is to exploit certain lexico-syntactic patterns to detect is-a relationships in natural language. For instance, patterns like “NPy such as NPx” or “NPx and other NPy” often indicate a hypernymy relationship $(u,\texttt{is-a},v)$. By treating unique noun phrases as nodes in a large, directed graph, we may construct a Hearst Graph using only unstructured text and very limited prior knowledge in form of patterns. Table 1 lists the only patterns that we use in this work. Formally, let ${E=\{(u,v)\}_{i=1}^{N}}$ denote the set of is-a relationships that have been extracted from a text corpus. Furthermore, let $w(u,v)$ denote how often we have extracted the relationship $u,\texttt{is-a},v$. We then represent the extracted patterns as a weighted directed graph $G=(V,E,w)$ where $V$ is the set of all extracted terms.

Hearst patterns afford a number of important advantages in terms of data acquisition: they are embarrassingly parallel across both sentences and distinct Hearst patterns, and counts are easily aggregated in any MapReduce setting Dean and Ghemawat (2004). Our own experiments, and those of Seitner et al. (2016), demonstrate that this approach can be scaled to large corpora such as CommonCrawl.111http://commoncrawl.org As Roller et al. (2018) showed, pattern matches also provide important contextual constraints which boost signal compared to methods based on the Distributional Inclusion Hypothesis.

However, naïvely using Hearst pattens can easily result in a graph that is extremely sparse: pattern matches naturally follow a long-tailed distribution that is skewed by the occurrence probabilities of constituent words (see Figure 2) and many true relationships are unlikely to ever appear in a corpus (e.g. “long-tailed macaque is-a entity”). This may be alleviated with generous, low precision patterns Seitner et al. (2016), but the resulting graph will contain many false positives, inconsistencies, and cycles. For example, our own Hearst graph contains the cycle: (area, is-a, spot), (spot, commercial), (commercial, promotion), (promotion, area), which is caused by the polysemy of spot (location, advertisement) and area (location, topical area).

3.2 Hyperbolic Embeddings

Roller et al. (2018) showed that low-rank embedding methods, such as Singular Value Decomposition (SVD), alleviate the aforementioned sparsity issues but still produce cyclic and inconsistent predictions. In the following, we will discuss how hyberbolic embeddings allow us to improve consistency via strong hierarchical priors in the geometry.

First, we will briefly review necessary concepts of hyperbolic embeddings: In contrast to Euclidean or Spherical space, there exist multiple equivalent models for hyperbolic space.222e.g., the Poincaré ball, the Lorentz model, the Poincaré upper half plane, and the Beltrami-Klein model Since there exist transformations between these models that preserve all geometric properties (including isometry), we can choose whichever is best suited for a given task. In the following, we will first discuss hyperbolic embeddings based on the Poincaré-ball model, which is defined as follows: The Poincaré-ball model is the Riemannian manifold ${\mathbb{P}^{n}=(\mathbb{B}^{n},d_{p})}$, where $\mathbb{B}^{n}=\{\bm{x}\in\mathbb{R}^{n}:\|\bm{x}\|<1\}$ is the open $n$-dimensional unit ball and where $d_{p}$ is the distance function

[TABLE]

Hyperbolic space has a natural hierarchical structure and, intuitively, can be thought of as a continuous versions of trees. This property becomes evident in the Poincaré ball model: it can be seen from Equation 1, that the distance within the Poincaré ball changes smoothly with respect to the norm of a point $\bm{u}$. Points that are close to the origin of the disc are relatively close to all other points in the ball, while points that are close to the boundary are relatively far apart333This can be seen by considering how the Euclidean distance in $\delta(\bm{u},\bm{v})$ is scaled by the norms of the respective points. This locality property of the distance is key for learning continuous embeddings of hierarchical structures and corresponds to the behavior of shortest-paths in trees.

Hyperbolic Entailment Cones

Ganea et al. (2018) used the hierarchical properties of hyperbolic space to define entailment cones in an embedding. The main idea of hyperbolic entailment cones (HECs) is to define for each possible point $\bm{v}$ in the space, an entailment region in the form of a hyperbolic cone $\mathcal{C}_{\bm{v}}$. Points $\bm{u}$ that are located inside a cone $\mathcal{C}_{\bm{v}}$ are assumed to be children of $\bm{v}$. The width of each cone is determined by the norm $\|\bm{v}\|$ of the associated base point. The closer $\bm{v}$ is to the origin, i.e., the more general the basepoint is, the larger the width of $\mathcal{C}_{\bm{v}}$ becomes and the more points are subsumed in the entailment cone. Equation 3 shows entailment cones for different points in $\mathbb{P}^{2}$. To model the possible entailment $(u,\texttt{is-a},v)$, we can then use the energy function

[TABLE]

In Equation 2, $\alpha(\bm{v})$ denotes the half-aperture of the cone associated with point $\bm{v}$ and $\phi(\bm{v},\bm{u})$ denotes the angle between the half-lines $(\bm{v}\bm{u}$ and $(0\bm{v}$. If $u\in\mathcal{C}_{\bm{v}}$, i.e., if the angle between $\bm{u}$ and $\bm{v}$ is smaller than the half aperture of $\mathcal{C}_{\bm{v}}$, it holds that $\mathcal{E}(u,v)=0$. If $u\not\in\mathcal{C}_{\bm{v}}$, the energy $\mathcal{E}(u,v)$ can then be interpreted as smallest angle of a rotation bringing $\bm{u}$ into the cone associated with $\bm{v}$.

Given a Hearst graph $G=(V,E,w)$, we then compute embeddings of all terms in the following way: Let $\bm{v}_{i}$ denote the embedding of term $i$ and let $\Theta=\{\bm{v}_{i}\}_{i=1}^{M}$. To minimize the overall energy of the embedding, we solve the optimization problem

[TABLE]

where

[TABLE]

The goal of Equation 3 is therefore to find a joint embedding of all terms that best explains all observed Hearst patterns.

Lorentz Entailment Cones

The optimization problem Equation 3 is agnostic of the hyperbolic manifold on which the optimization is performed. Ganea et al. (2018) developed hyperbolic entailment cones in the Poincaré-ball model. However, as Nickel and Kiela (2018) pointed out, the Poincaré-ball model is not optimal from an optimization perspective as it is prone to numerical instabilities when points approach the boundary of the ball. Instead, Nickel and Kiela (2018) proposed to perform optimization in the Lorentz model and use the Poincaré ball only for analysis and visualization. Here, we follow this approach and develop entailment cones in the Lorentz model of hyperbolic space. The Lorentz model is defined as follows: let $\bm{x}$, $\bm{y}\in\mathbb{R}^{n+1}$ and let

[TABLE]

denote the Lorentzian scalar product. The Lorentz model of $n$-dimensional hyperbolic space is then the Riemannian manifold ${\mathbb{L}^{n}=(\mathbb{H}^{n},d_{\ell})}$, where

[TABLE]

denotes the upper sheet of a two-sheeted $n$-dimensional hyperboloid and where the associated distance function on $\mathbb{L}$ is given as

[TABLE]

Due to the equivalence of both models, we can define a mapping between both spaces that preserves all properties including isometry. Points in the Lorentz model can be mapped into the Poincaré ball via the diffeomorphism $p:\mathbb{H}^{n}\to\mathbb{P}^{n}$, where

[TABLE]

Furthermore, points in $\mathbb{P}^{n}$ can be mapped to $\mathbb{L}^{n}$ via

[TABLE]

See also Figure 4 for an illustration of the Lorentz model and its connections to the Poincaré ball.

To define entailment cones in the Lorentz model, it is necessary to derive $\alpha(\bm{v})$ and $\phi(\bm{u},\bm{v})$ in $\mathbb{L}^{n}$. Both quantities can be derived easily from the hyperbolic law of cosines and the mapping between $\mathbb{P}^{n}$ and $\mathbb{L}^{n}$. In particular, it holds that

[TABLE]

Due to space restrictions, we refer to the supplementary material for the full derivation.

Training

To solve Equation 3, we follow Nickel and Kiela (2018) and perform stochastic optimization via Riemannian SGD (RSGD; Bonnabel 2013). In RSGD, updates to the parameters $\theta$ are computed via

[TABLE]

where $\operatorname{grad}f(\theta_{t})$ denotes the Riemannian gradient and $\eta$ denotes the learning rate. In Equation 5, the Riemannian gradient of $f$ at $\theta$ is computed via

[TABLE]

where $\nabla f(\theta)$ denotes the Euclidean gradient of $f$ and where

[TABLE]

denote the projection from the ambient space $\mathbb{R}^{n+1}$ onto the tangent space $\mathcal{T}_{\bm{x}}\mathbb{L}^{n}$ and the inverse of the metric tensor, respectively. Finally, the exponential map for $\mathbb{L}^{n}$ is computed via

[TABLE]

where $\|\bm{v}\|_{\mathbb{L}}=\sqrt{\langle\bm{v},\bm{v}\rangle_{\mathbb{L}}}$ and $\bm{v}\in\mathcal{T}_{\bm{x}}\mathbb{L}^{n}$.

As suggested by Nickel and Kiela (2018), we initialize the embeddings close to the origin of $\mathbb{L}^{n}$ by sampling from the uniform distribution $\mathcal{U}(-0.001,0.001)$ and by setting $x_{0}$ to $\sqrt{1+||x^{\prime}||^{2}}$, what ensures that the sampled points are located on the surface of the hyperboloid.

4 Experiments

To evaluate the efficacy of our method, we evaluate on several commonly-used hypernymy benchmarks (as described in Roller et al. (2018)) as well as in a reconstruction setting (as described in Nickel and Kiela (2017)). Following Roller et al. (2018), we compare to the following methods for unsupervised hypernymy detection:

Pattern-Based Models

Let $E=\{(x,y)\}_{i=1}^{N}$ be the set of Hearst patterns in our corpus, $w(x,y)$ be the count of how many times $(x,y)$ occurs in $E$, and $W=\sum_{(x,y)\in E}w(x,y)$. We then consider the following pattern-based methods:

Count Model (p)

This models simply outputs the count, or equivalently, the extraction probabilities of Hearst patterns, i.e.,

[TABLE]

PPMI Model (ppmi)

To correct for skewed occurrence probabilities, the PPMI model predicts hypernymy relations based on the Pointwise Mutual Information over the Hearst pattern corpus. Let $p^{-}(x)=\Sigma_{(x,y)\in\mathcal{P}}w(x,y)/W$ and ${p^{+}(x)=\Sigma_{(y,x)\in E}w(y,x)}$, then:

[TABLE]

SVD Count (sp)

To account for missing relations, we also compare against low-rank embeddings of the Hearst corpus using Singular Value Decomposition (SVD). Specifically, let $X\in R^{MxM}$, such that $X_{ij}=w(i,j)/W$ and $U\Sigma V^{\top}$ be the singular value decomposition of $X$, then:

[TABLE]

SVD PPMI (spmi)

We also evaluate against the SVD of the PPMI matrix, which is identical to $\text{sp}(i,j)$, with the exception that $X_{ij}=\text{ppmi}(i,j)$, instead of $p(i,j)$. Roller et al. (2018) showed that this method provides state-of-the-art results for unsupervised hypernymy detection.

Hyperbolic Embeddings (HypeCone)

We embed the Hearst graph into hyperbolic space as described in section 3.2. At evaluation time, we predict the likelihood using the model energy $\mathcal{E}(u,v)$.

Distributional Models

The distributional models in our evaluation are based on the DIH, i.e., the idea that contexts in which a narrow term $x$ may appear (ex: cat) should be a subset of the contexts in which a broader term $y$ (ex: animal) may appear.

WeedsPrec

The first distributional model we consider is WeedsPrec Weeds et al. (2004), which captures the features of $x$ which are included in the set of more general term’s features, $y$:

[TABLE]

invCL

Lenci and Benotto (2012), introduce the idea of distributional exclusion by also measuring the degree to which the broader term contains contexts not used by the narrower term. The degree of inclusion is denoted as:

[TABLE]

To measure the inclusion of $x$ and $y$ and the non-inclusion of $y$ in $x$, invCL is then computed as

[TABLE]

SLQS

The SLQS model is based on the informativeness hypothesis Santus et al. (2014); Shwartz et al. (2017), i.e., the idea that general words appear mostly in uninformative contexts, as measured by entropy. SLQS depends on the median entropy of a term’s top $k$ contexts:

[TABLE]

where $H(c_{i})$ is the Shannon entropy of context $c_{i}$ across all terms. SLQS is then defined as:

[TABLE]

Corpora and Preprocessing

We construct our Hearst graph using the same data, patterns, and procedure as described in Roller et al. (2018): Hearst patterns are extracted from the concatenation of GigaWord and Wikipedia. The corpus is tokenized, lemmatized, and POS-tagged using CoreNLP 3.8.0 Manning et al. (2014). The full set of Hearst patterns is provided in Table 1. These include prototypical Hearst patterns, like “animals [such as] big cats,” as well as broader patterns like “New Year [is the most important] holiday.” Noun phrases were allowed to match limited modifiers, and produced additional hits for the head of the noun phrase. The final corpus contains circa 4.5M matched pairs, 431K unique pairs, and 243K unique terms.

Hypernymy Tasks

We consider three distinct subtasks for evaluating the performance of these models for hypernymy prediction:

• •

Detection: Given a pair of words ($u$, $v$), determine if $v$ is a hypernym of $u$.

• •

Direction: Given a pair ($u$, $v$), determine if $u$ is more general than $v$ or vise versa.

• •

Graded Entailment: Given a pair of words ($u$, $v$), determine the degree to which $u$ is a $v$.

For detection, we evaluate all models on five commonly-used benchmark datasets: Bless Baroni and Lenci (2011), Leds Baroni et al. (2012), Eval Santus et al. (2015), Shwartz Shwartz et al. (2016), and WBless Weeds et al. (2014), In addition to positive hypernymy relations, these datasets includes negative samples in the form of random pairs, co-hyponymy, antonymy, meronymy, and adjectival relations. For directionality and graded entailment, we also use the BiBless (Kiela et al., 2015) and Hyperlex (Vulic et al., 2016) datasets. We refer to Roller et al. (2018) for an in-depth discussion of these datasets.

Table 2 shows the results for all tasks on these datasets. It can be seen that our proposed approach provides substantial gains on the detection and directionality tasks and, overall, achieves state of the art results on seven of these nine benchmarks. In addition, our method clearly outperforms other embedding-based approaches on Hyperlex, although it can not fully match the performance of the count-based methods. As Roller et al. (2018) noted, this might be an artifact of the evaluation metric, as count-based methods benefit from their sparse-predictions in this setting.

It can also be seen that our method outperforms Poincaré GloVe for the task of hypernymy prediction. While Tifrea et al. (2018) report $0.341$ Spearman’s $\rho$ on HyperLex and $0.652$ accuracy on WBless, our method can achieve substantially better results for the same tasks ($\rho=0.59$ on HyperLex, $\text{ACC}=0.909$ on WBLess). This illustrates the importance of the distributional constraints that are provided by the Hearst patterns.

An additional benefit is the efficiency of our embedding. For all tasks, we have used a 20-dimensional embedding for HypeCones, while the best results for SVD-based methods have been achieved with 300 dimensions. This reduction in parameters by over an order of magnitude clearly highlights the efficiency of hyperbolic embeddings for representing hierarchical structures.

Reconstruction

In the following, we compare embedding and pattern-based methods on the task of reconstructing an entire subtree of WordNet, i.e., the animals, plants, and vehicles taxonomies, as proposed by Kozareva and Hovy (2010). In addition to predicting the existence of single hypernymy relations, this allows us to evaluate the performance of these models for inferring full taxonomies and to perform an ablation for the prediction of missing and transitive relations. We follow previous work (Bordes et al., 2013; Nickel and Kiela, 2017) and report for each observed relation $(u,v)$ in WordNet, its score ranked against the score of the ground-truth negative edges. In Table 3, All refers to the ranking of all edges in the subtree, Missing to edges that are not included in the Hearst graph $G$, Transitive to missing transitive edges in $G$ (i.e. for all edges ${\{((x,z):(x,y),(y,z)\in E\land(x,z)\notin E\}}$).

It can be seen that our method clearly outperforms the SVD and count-based models with a relative improvement of typically over $40\%$ over the best non-hyperbolic model. Furthermore, our ablation shows that HypeCones improves the consistency of the embedding due to its transitivity property. For instance, in our Hearst Graph the relation (male horse, is-a, equine) is missing. However, since we correctly model that (male horse, is-a, horse) and (horse, is-a, equine), by transitivity, we also infer (male horse, is-a, equine), which SVD fails to do.

5 Conclusion

In this work, we have proposed a new approach for inferring concept hierarchies from large text corpora. For this purpose, we combine Hearst patterns with hyperbolic embeddings what allows us to set appropriate constraints on the distributional contexts and to improve the consistency in the embedding space. By computing a joint embedding of all terms that best explains the extracted Hearst patterns, we can then exploit these properties for improved hypernymy prediction. The natural hierarchical structure of hyperbolic space allows us also to learn very efficient embeddings that reduce the required dimensionality substantially over SVD-based methods. To improve optimization, we have furthermore proposed a new method to compute entailment cones in the Lorentz model of hyperbolic space. Experimentally, we show that our embeddings achieve state-of-the-art performance on a variety of commonly-used hypernymy benchmarks.