Low-rank approximations of hyperbolic embeddings

TL;DR

This paper introduces a method for learning hyperbolic embeddings constrained to low-rank subspaces, enabling efficient modeling of hierarchical data with shared structures.

Contribution

It proposes a novel low-rank factorization approach for hyperbolic embeddings using manifold optimization techniques, which is computationally efficient and effective.

Findings

Effective low-rank hyperbolic embeddings learned

Improved modeling of hierarchical structures

Efficient algorithms demonstrated

Abstract

The hyperbolic manifold is a smooth manifold of negative constant curvature. While the hyperbolic manifold is well-studied in the literature, it has gained interest in the machine learning and natural language processing communities lately due to its usefulness in modeling continuous hierarchies. Tasks with hierarchical structures are ubiquitous in those fields and there is a general interest to learning hyperbolic representations or embeddings of such tasks. Additionally, these embeddings of related tasks may also share a low-rank subspace. In this work, we propose to learn hyperbolic embeddings such that they also lie in a low-dimensional subspace. In particular, we consider the problem of learning a low-rank factorization of hyperbolic embeddings. We cast these problems as manifold optimization problems and propose computationally efficient algorithms. Empirical results illustrate the efficacy of the proposed approach.

Figures

Tables

Table 1. Table 1: Mean average precision (MAP) score obtained by the proposed approaches on the mammal dataset.

Rank	Method-1	Method-2	Method-3
$5$	$0.8274$	$0.8416$	$0.1516$
$10$	$0.9106$	$0.9143$	$0.9004$
$20$	$0.9388$	$0.9390$	$0.9388$
$50$	$0.9488$	$0.9486$	$0.9488$
$100$	$0.9504$	$0.9504$	$0.9504$
$200$	$0.9502$	$0.9502$	$0.9502$
$300$	$0.9501$	$0.9501$	$0.9501$

Table 2. Table 2: Mean average precision (MAP) score obtained by the proposed approaches on the noun dataset.

Rank	Method-1	Method-2	Method-3
$5$	$0.5343$	$0.5422$	$0.5343$
$10$	$0.6742$	$0.6796$	$0.6742$
$20$	$0.7425$	$0.7449$	$0.7425$
$50$	$0.7887$	$0.7891$	$0.7887$
$100$	$0.8070$	$0.8070$	$0.8070$

Equations

$${\mathbb{B}}^n=\{\mathbf{u}\in {\mathbb{R}}^n|\Vert \mathbf{u}\Vert <1\},$$

$$d_{\mathbb{B}}(\mathbf{u},\mathbf{v})=\mathrm{arccosh}\left(1+2\frac{{\Vert \mathbf{u}-\mathbf{v}\Vert }^2}{(1-{\Vert \mathbf{u}\Vert }^2)(1-{\Vert \mathbf{v}\Vert }^2)}\right)$$

$${\Vert \mathbf{u}\Vert }_{\mathbb{B}}:=d_{\mathbb{B}}(0,\mathbf{u})=2\mathrm{arctanh}(\Vert \mathbf{u}\Vert ).$$

$${⟨\bar{\mathbf{u}},\bar{\mathbf{v}}⟩}_{\mathcal{L}}={\bar{\mathbf{u}}}^{\top }\mathbf{L}\bar{\mathbf{v}}=-u_0{v}_0+{\mathbf{u}}^{\top }\mathbf{v},$$

$$\mathbf{L}=\left[\begin{array}{cc}-1& 0_n^{\top }\\ 0_n& {\mathbf{I}}_n\end{array}\right],$$

$${\mathbb{H}}^n=\{\bar{\mathbf{u}}\in {\mathbb{R}}^{n+1}|{⟨\bar{\mathbf{u}},\bar{\mathbf{u}}⟩}_{\mathcal{L}}=-1,u_0>0\}.$$

$$d_{\mathbb{H}}(\bar{\mathbf{u}},\bar{\mathbf{v}})=\mathrm{arccosh}(-{⟨\bar{\mathbf{u}},\bar{\mathbf{v}}⟩}_{\mathcal{L}}).$$

$$h:{\mathbb{H}}^n\to {\mathbb{B}}^n,h(\bar{\mathbf{u}})=\frac{\mathbf{u}}{u_0+1},\bar{\mathbf{u}}\in {\mathbb{H}}^n.$$

$$h^{-1}(\mathbf{w})=\frac{1}{(1-{\Vert \mathbf{w}\Vert }^2)}\left[\begin{array}{c}(1+{\Vert \mathbf{w}\Vert }^2)\\ 2\mathbf{w}\end{array}\right],\mathbf{w}\in {\mathbb{B}}^n.$$

$$\bar{\mathbf{X}}=\left[\begin{array}{c}{\mathbf{x}}_0\\ \mathbf{X}\end{array}\right]\text{ and }{\bar{\mathbf{x}}}_i=\left[\begin{array}{c}{x}_{0i}\\ {\mathbf{x}}_i\end{array}\right].$$

$${\bar{\mathbf{x}}}_i=\left[\begin{array}{c}{x}_{0i}\\ {\mathbf{x}}_i\end{array}\right]\approx \left[\begin{array}{c}{z}_{0i}\\ \mathbf{U}{\mathbf{z}}_i\end{array}\right]={\hat{\mathbf{z}}}_i\forall i=1,\dots ,m,$$

$$-z_{0i}^2+(\mathbf{U}{\mathbf{z}}_i{)}^{\top }(\mathbf{U}{\mathbf{z}}_i)=-1.$$

$$\underset{\begin{array}{c}\mathbf{U}\in {\mathbb{R}}^{n\times r},\\ {\bar{\mathbf{z}}}_i\in {\mathbb{R}}^{n+1}\forall i\end{array}}{\min }$$

$${\mathbf{U}}^{\top }\mathbf{U}={\mathbf{I}}_r,{\bar{\mathbf{z}}}_i\in {\mathbb{H}}^r,$$

$${\hat{\mathbf{z}}}_i=\left[\begin{array}{cc}1& 0_r^{\top }\\ 0_n& \mathbf{U}\end{array}\right]{\bar{\mathbf{z}}}_i\forall i=1,\dots ,m,$$

$$\begin{array}[]{lll}\mathop{\rm min}\nolimits\limits_{y\in\mathcal{M}}\quad f(y),\end{array}$$

$$\begin{array}[]{lll}T_{y}\mathcal{M}:=T_{\mathbf{U}}{\mathrm{St}({r},{n})}\times T_{\bar{{\mathbf{z}}}_{1}}\mathbb{H}^{r}\times\ldots\times T_{\bar{{\mathbf{z}}}_{m}}\mathbb{H}^{r}\\ =\{(\xi_{{\mathbf{U}}},\xi_{\bar{{\mathbf{z}}}_{i}},\ldots,\xi_{\bar{{\mathbf{z}}}_{m}}):{\mathrm{symm}}({\mathbf{U}}^{\top}\xi_{{\mathbf{U}}})={\mathbf{0}}_{r}\ \text{ and }\\ \quad\quad\left\langle{\bar{{\mathbf{z}}}}_{i},\xi_{\bar{{\mathbf{z}}}_{i}}\right\rangle_{\mathcal{L}}=0\text{ for all }i\},\end{array}$$

$$\begin{array}[]{llll}g_{y}(\xi_{y},\eta_{y})\coloneqq\langle\xi_{{\mathbf{U}}},\eta_{{\mathbf{U}}}\rangle+\ \sum_{i=1}^{m}\left\langle\xi_{\bar{{\mathbf{z}}}_{i}},\eta_{\bar{{\mathbf{z}}}_{i}}\right\rangle_{\mathcal{L}},\end{array}$$

$${\Pi }_y({\zeta }_{\mathbf{U}},{\zeta }_{{\bar{\mathbf{z}}}_i},\dots ,{\zeta }_{{\bar{\mathbf{z}}}_m}):=$$

$${\zeta }_{{\bar{\mathbf{z}}}_i}+{\bar{\mathbf{z}}}_i{⟨{\mathbf{z}}_i,{\zeta }_{{\bar{\mathbf{z}}}_i}⟩}_{\mathcal{L}},$$

$$\dots ,$$

$${\zeta }_{{\bar{\mathbf{z}}}_m}+{\bar{\mathbf{z}}}_m{⟨{\mathbf{z}}_m,{\zeta }_{{\bar{\mathbf{z}}}_m}⟩}_{\mathcal{L}}).$$

$$R_y({\xi }_y):=$$

$${\bar{\mathbf{z}}}_i\cosh (\Vert {\xi }_{{\bar{\mathbf{z}}}_i}{\Vert }_{\mathcal{L}})+{{\xi }_{\bar{\mathbf{z}}}}_i\sinh (\Vert {\xi }_{{\bar{\mathbf{z}}}_i}{\Vert }_{\mathcal{L}})/\Vert {\xi }_{{\bar{\mathbf{z}}}_i}{\Vert }_{\mathcal{L}},$$

$$\dots ,$$

$${\bar{\mathbf{z}}}_m\cosh (\Vert {\xi }_{{\bar{\mathbf{z}}}_m}{\Vert }_{\mathcal{L}})+{{\xi }_{\bar{\mathbf{z}}}}_m\sinh (\Vert {\xi }_{{\bar{\mathbf{z}}}_m}{\Vert }_{\mathcal{L}})/\Vert {\xi }_{{\bar{\mathbf{z}}}_m}{\Vert }_{\mathcal{L}}),$$

$${\mathrm{grad}}_{y}f={\Pi }_y({\nabla }_{y}f),\text{ and }$$

$${\mathrm{Hess}}_{y}f[{\xi }_y]={\Pi }_y(\mathrm{D}{\mathrm{grad}}_{y}f[{\xi }_y]),$$

Full text

1 Introduction

Learning hyperbolic representation of entities have gained recent interest in the machine learning community [1, 2, 3, 4, 5]. In particular, hyperbolic embeddings have been shown to be well-suited for various natural language processing problems [3, 6, 7] that require modeling hierarchical structures such as knowledge graphs, hypernymy hierarchies, organization hierarchy, and taxonomies, among others. The reason being learning representations in the hyperbolic space provides a principled approach for integrating structural information encoded in such (discrete) entities into continuous space.

The hyperbolic space is a non-Euclidean space and has constant negative curvature. The latter property enables it to grow exponentially even in dimension as low as two. Hence, the hyperbolic space has been considered to model trees and complex networks, among others [8, 9]. Figure 1(a) is an example of representing a part of mammal taxonomy tree in a hyperbolic space (two-dimensional Poincaré ball). Hyperbolic embeddings (numerical representations of tasks) have been considered in several applications such as question answering system [10], recommender systems [11, 12], link prediction [13, 14], natural language inference [15], vertex classification [16], and machine translation [17].

In this paper, we consider the setting in which an additional low-rank structure may also exist among the learned hyperbolic embeddings. Such a setting may arise when the hierarchical entities are closely related. We propose to learn a low-rank approximation of the given (high dimensional) hyperbolic embeddings. Conceptually, we model high dimensional hyperbolic embeddings as a product of a low-dimensional subspace and low-dimensional hyperbolic embeddings. The optimization problem is cast on the product of the Stiefel and hyperbolic manifolds. We develop an efficient Riemannian trust-region algorithm for solving it. We evaluate the proposed approach on real-world datasets: on the problem of reconstructing taxonomy hierarchies from the embeddings. We observe that the performance of proposed approach match the original embeddings even in low-rank settings.

The outline of the paper is as follows. Section 2 discusses two popular models of representing hyperbolic space in the Euclidean setting. In Section 3, we present our formulation to approximate given hyperbolic embeddings in a low-rank setting. The optimization algorithm is discussed in Section 4. The experimental results are presented in Section 5 and Section 6 concludes the paper.

2 Background

In this section, we briefly discuss the basic concepts of hyperbolic geometry. Interested readers may refer [18, 19] for more details.

The hyperbolic space (of dimension $\geq 2$) is a Riemannian manifold with a constant negative sectional curvature. Similar to the Euclidean or spherical spaces, it is isotropic. However, the Euclidean space is flat (zero curvature) and the spherical space is positively curved. As a result of negative curvature, the circumference and area of a circle in hyperbolic space grow exponentially with the radius. In contrast, the circumference and area of a circle in the hyperbolic space grow linearly and quadratically, respectively, in Euclidean setting. Hence, hyperbolic spaces expand faster than the Euclidean spaces. Informally, hyperbolic spaces may be viewed as a continuous counterpart to discrete trees as the metric properties of a two-dimensional hyperbolic space and a $b$-ary tree (a tree with branching factor $b$) are similar. Hence, trees can be embedded into a two-dimensional hyperbolic space while keeping the overall distortion arbitrarily small. In contrast, Euclidean spaces cannot attain this result even with unbounded number of dimensions.

Since hyperbolic models cannot be represented within Euclidean space without distortion, several (equivalent) models exist for representing hyperbolic spaces for computation purpose. The models are conformal to the Euclidean space and points in one model can be transformed to be represented in another model, while preserving geometric properties such as isometry. However, no model captures all the properties of the hyperbolic geometry. Two hyperbolic models, in particular, have received much interest recently in the machine learning community: the Poincaré ball model and the hyperboloid model.

2.1 Poincaré ball model

The Poincaré ball is a $n$-dimensional hyperbolic space defined as the interior of the $n$-dimensional unit (Euclidean) ball:

[TABLE]

where $\left\|\cdot\right\|$ denotes the Euclidean norm. The distance between two points $\mathbf{u},\mathbf{v}\in\mathbb{B}^{n}$ in the Poincaré ball model is given by

[TABLE]

and the Poincaré norm is given by

[TABLE]

We observe that the distance between a pair of points near the boundary of the Poincaré ball (Euclidean norm close to unity) grows much faster than distance between the points close to the center (Euclidean norm close to zero). In addition, the distance within the Poincaré ball varies smoothly with respect to points $\mathbf{u}$ and $\mathbf{v}$. These properties are helpful embedding discrete hierarchical structures such as trees in hyperbolic spaces and obtain continuous embeddings which respect the tree metric structure. For instance, the origin of the Poincaré ball may be mapped to the root node of the tree as the root node is relatively closer to all other nodes (points). The leaf nodes can be places near the boundary to ensure they are relatively distant from other leaf nodes. Additionally, the shortest path between a pair of points is usually via a point closer to the origin, just as the shortest path between two nodes in a tree is via their parent nodes. Figure 1(b) shows a Poincaré disk ($\mathbb{B}^{2}$) embedding a tree with two regular subtrees.

2.2 Hyperboloid model

Let $\bar{\mathbf{u}},\bar{\mathbf{v}}\in\mathbb{R}^{n+1}$ such that $\bar{\mathbf{u}}=\begin{bmatrix}u_{0}\\ \mathbf{u}\end{bmatrix}$ and $\bar{\mathbf{v}}=\begin{bmatrix}v_{0}\\ \mathbf{v}\end{bmatrix}$ and $\mathbf{u},\mathbf{v}\in\mathbb{R}^{n}$. The Lorentz scalar product $\left\langle\cdot,\cdot\right\rangle_{\mathcal{L}}$ of two vectors $\bar{\mathbf{u}}$ and $\bar{\mathbf{v}}$ is defined as

[TABLE]

where $\mathbf{L}$ is a $(n+1)$-dimensional diagonal matrix

[TABLE]

${\mathbf{0}}_{n}$ is the $n$-dimensional zero column vector, and ${\mathbf{I}}_{n}$ is the $n$-dimensional identity matrix.

The hyperboloid model, also known as the Lorentz model of hyperbolic geometry, is given by

[TABLE]

The model represents the upper sheet of an $n$-dimensional hyperboloid. From the constraint set, it can be observed that if $\bar{\mathbf{u}}\in\mathbb{H}^{n}$, then $u_{0}=\sqrt{1+\mathbf{u}^{\top}\mathbf{u}}$.

The distance between two points $\bar{\mathbf{u}},\bar{\mathbf{v}}\in\mathbb{H}^{n}$ in the hyperboloid model is given by

[TABLE]

As stated earlier, both the Poincaré ball and the hyperboloid models are equivalent and a mapping exists from one model to another [21]. Points on the hyperboloid can be mapped to the Poincaré ball by

[TABLE]

The reverse mapping, $h^{-1}:\mathbb{B}^{n}\rightarrow\mathbb{H}^{n}$ is defined as follows:

[TABLE]

Figure 1(c) shows a two-dimensional hyperboloid model $\mathbb{H}^{2}$. It can be observed that the Poincaré disk $\mathbb{B}^{2}$ is obtained as a stereoscopic projection of $\mathbb{H}^{2}$.

3 Low-rank parameterization in

hyperbolic space

As discussed earlier, hyperbolic embeddings are typically suitable for representing elements of hierarchical structures such as nodes of trees [3] and complex networks [8] to name a few. When the task involves closely related hierarchical concepts, additional low-rank structure may also exist among such hyperbolic embeddings. In this section, we propose a novel low-rank parameterization for hyperbolic embeddings. It should be noted that, unlike the Euclidean embeddings, incorporating a low-rank structure in the hyperbolic framework is non-trivial because of the hyperboloid constraints.

Let $\bar{{\mathbf{X}}}$ be a $(n+1)\times m$ matrix whose columns represent $n$-dimensional hyperbolic embeddings corresponding to $m$ elements from a given hierarchical structure. For notational convenience, we represent $\bar{{\mathbf{X}}}$ and its $i$-th column $\bar{{\mathbf{x}}}_{i}$ as follows:

[TABLE]

We propose to approximate $\bar{{\mathbf{x}}}_{i}$ as a low-dimensional hyperbolic embedding such that ${\mathbf{x}}_{i}$ shares a latent low-dimensional subspace with ${\mathbf{x}}_{j}$ (corresponding to $\bar{{\mathbf{x}}}_{j}$), for all $i,j=1,\ldots,m$. Mathematically, we propose the following $(r+1)$-rank approximation for $\bar{{\mathbf{x}}}_{i}$:

[TABLE]

where $\hat{{\mathbf{z}}}_{i}\in\mathbb{H}^{n}$, ${\mathbf{z}}_{i}\in\mathbb{R}^{r}$, ${\mathbf{U}}\in\mathbb{R}^{n\times r}$, and ${\mathbf{U}}^{\top}{\mathbf{U}}={\mathbf{I}}_{r}$. We discuss below the consequences of the proposed model.

Firstly, we obtain $\bar{{\mathbf{z}}}_{i}=\begin{bmatrix}z_{0i}\\ {\mathbf{z}}_{i}\end{bmatrix}\in\mathbb{H}^{r}$. This is because $\hat{{\mathbf{z}}}_{i}\in\mathbb{H}^{n}$ implies

[TABLE]

$\bar{{\mathbf{z}}}_{i}\in\mathbb{H}^{r}$ follows from the above equality as ${\mathbf{U}}^{\top}{\mathbf{U}}={\mathbf{I}}_{r}$. Secondly, the matrix ${\mathbf{X}}$ (corresponding to $\bar{{\mathbf{X}}}$) is modeled as a low-rank matrix as we approximate ${\mathbf{X}}$ as ${\mathbf{U}}{\mathbf{Z}}$, where ${\mathbf{Z}}=[{\mathbf{z}}_{1},\ldots,{\mathbf{z}}_{m}]$. Thirdly, the space complexity of embeddings reduces from $O(nm)$ (for $\bar{{\mathbf{X}}}$) to $O(nr+mr)$ (for ${\mathbf{U}},{\mathbf{Z}}$ and ${\mathbf{z}}_{0}$).

We propose to learn the proposed low-rank paramterization of $\bar{{\mathbf{X}}}$ by solving the optimization problem:

[TABLE]

where $\ell:\mathbb{R}^{n+1}\times\mathbb{R}^{n+1}\rightarrow[0,\infty)$ is a loss function that measures the quality of the proposed approximation. Let function $f$ denote the objective function in (4), i.e., $f({\mathbf{U}},\bar{{\mathbf{Z}}})=\sum_{i=1}^{m}\ell(\bar{{\mathbf{x}}_{i}},\hat{{\mathbf{z}}}_{i})$.

We discuss the following three choices of $f$:

$f({\mathbf{U}},\bar{{\mathbf{Z}}};\bar{{\mathbf{X}}})=\|{\mathbf{X}}-{\mathbf{U}}{\mathbf{Z}}\|_{F}^{2}$:

we penalize the Euclidean distance between ${\mathbf{X}}$ and ${\mathbf{U}}{\mathbf{Z}}$. This is because ${\mathbf{x}}_{0}$ and ${\mathbf{z}}_{0}$ are determined from the hyperboloid constraint given ${\mathbf{X}}$ and ${\mathbf{U}}{\mathbf{Z}}$, respectively. We obtain a closed-form solution of (4) with this loss function and the solution involves computing a rank-$r$ singular value decomposition of ${\mathbf{X}}$. In Section 5, we denote this approach by the term Method-1. 2. 2.

$f({\mathbf{U}},\bar{{\mathbf{Z}}};\bar{{\mathbf{X}}})=\|\bar{{\mathbf{X}}}-\hat{{\mathbf{Z}}}\|_{F}^{2}$:

we penalize the Euclidean distance between the (full) hyperbolic embeddings (matrices) $\bar{{\mathbf{X}}}$ and $\hat{{\mathbf{Z}}}=\begin{bmatrix}1&{\mathbf{0}}_{r}^{\top}\\ {\mathbf{0}}_{n}&{\mathbf{U}}\end{bmatrix}\bar{{\mathbf{Z}}}$. This approach is denoted by the term Method-2 in Section 5. 3. 3.

$f({\mathbf{U}},\bar{{\mathbf{Z}}};\bar{{\mathbf{X}}})=\sum_{i}\mathrm{arccosh}(-\left\langle\bar{{\mathbf{x}}}_{i},\hat{{\mathbf{z}}}_{i}\right\rangle_{\mathcal{L}})^{2}$:

since the columns of $\bar{{\mathbf{X}}}$ and $\hat{{\mathbf{Z}}}$ are hyperbolic embeddings, we penalize the hyperbolic distance (3) between the corresponding embeddings. We denote it by the term Method-3.

It should be noted that the problem (LABEL:eqn:generalOptimizationf) is a nonlinear and non-convex optimization problem, but has well-studied structured constraints. In particular, the structured constraints are cast has Riemannian manifolds. In the next section, we propose a Riemannian trust-region algorithm for solving (4) with the loss function discussed in options 2) and 3) above.

4 Optimization

It should be noted that the variable ${\mathbf{U}}$ in (4) belongs to the Stiefel manifold ${\mathrm{St}({r},{n})}\coloneqq\{{\mathbf{U}}\in\mathbb{R}^{n\times r}:{\mathbf{U}}^{\top}{\mathbf{U}}={\mathbf{I}}_{r}\}$ [22] and the variable $\bar{{\mathbf{z}}}_{i}$ belongs to the $r$-dimensional hyperbolic manifold $\mathbb{H}^{r}\coloneqq\{\bar{{\mathbf{z}}}_{i}\in\mathbb{R}^{r+1}:-z_{0i}^{2}+{\mathbf{z}}_{i}^{\top}{\mathbf{z}}_{i}=-1,z_{0i}>0\}$ for all $i=\{1,\ldots,m\}$. Consequently, the constraint set of the proposed optimization problem (4) is a smooth manifold $\mathcal{M}\coloneqq{\mathrm{St}({r},{n})}\times\mathbb{H}^{r}\times\ldots\times\mathbb{H}^{r}$, which is the Cartesian product of the Stiefel and $m$ hyperbolic manifolds of dimension $r$. The problem (4), therefore, now boils down to the manifold optimization problem:

[TABLE]

where $y$ has the representation $y\coloneqq({\mathbf{U}},\bar{{\mathbf{z}}}_{1},\ldots,\bar{{\mathbf{z}}}_{m})$ and $f:\mathcal{M}\rightarrow\mathbb{R}:y\mapsto f(y)=\sum_{i=1}^{n}\ell(\bar{{\mathbf{x}}_{i}},\hat{{\mathbf{z}}}_{i})$ is a smooth function.

We tackle the problem (5) in the Riemannian optimization framework that translates it into an unconstrained optimization problem over the nonlinear manifold $\mathcal{M}$, now endowed with a Riemannian geometry [23]. In particular, the Riemannian geometry on manifolds imposes a metric (inner product) structure on $\mathcal{M}$, which in turn allows to generalize notions like the shortest distance between points (on the manifold) or the translation of vectors on manifolds. Following this framework many of the standard nonlinear optimization algorithms in the Euclidean space, e.g., steepest descent and trust-regions, generalize well to Riemannian manifolds in a systematic manner. The Riemannian framework allows to develop computationally efficient algorithms on manifolds [23].

Both the Stiefel and hyperbolic manifolds are Riemannian manifolds, and their geometries have been individually well-studied in the literature [13, 23]. Subsequently, the manifold of interest $\mathcal{M}$ also has a Riemannian structure.

Below we list some of the basic optimization-related notions that are required to solve (5) with the Riemannian trust-region algorithm that exploits second-order information. The development of those notions follow the general treatment of manifold optimization discussed in [23, Chapter 7]. The Stiefel manifold related expressions follow from [23]. The hyperobolic related expressions follow from [13].

4.1 Metric and tangent space notions

Optimization on $\mathcal{M}$ is worked out on the tangent space, which is the linearization of $\mathcal{M}$ at a specific point. It is a vector space associated with each element of the manifold.

As $\mathcal{M}$ is a product space, its tangent space is also the product space of the tangent spaces of the Stiefel ${\mathrm{St}({r},{n})}$ and hyperbolic $\mathbb{H}^{r}$ manifolds. The characterization of the tangent space has the form:

[TABLE]

where $\mathrm{symm}$ extracts the symmetric part of a matrix.

As discussed above, to impose a Riemannian structure on $\mathcal{M}$, a smooth metric (inner product) definition is required at each element of the manifold. A natural choice of the metric $g_{y}:T_{y}\mathcal{M}\times T_{y}\mathcal{M}\rightarrow\mathbb{R}:(\xi_{y},\eta_{y})\mapsto g_{y}(\xi_{y},\eta_{y})$ on $\mathcal{M}$ is the summation of the individual Riemannian metrics on the Stiefel and hyperbolic manifolds. More precisely, we have

[TABLE]

where $\xi_{y}=(\xi_{{\mathbf{U}}},\xi_{\bar{{\mathbf{z}}}_{i}},\ldots,\xi_{\bar{{\mathbf{z}}}_{m}})$, $\eta_{y}=(\eta_{{\mathbf{U}}},\eta_{\bar{{\mathbf{z}}}_{i}},\ldots,\eta_{\bar{{\mathbf{z}}}_{m}})$, $\langle\cdot,\cdot\rangle$ is the standard inner product, and $\langle\cdot,\cdot\rangle_{\mathcal{L}}$ is the Lorentz inner product (2).

It should be emphasized that the metric in (7) endows the manifold $\mathcal{M}$ with a Riemannian structure and allows to develop various other notions of optimization in a straightforward manner.

One important ingredient required in optimization is the notion of an orthogonal projection operator $\Pi_{y}$ from the space $\mathbb{R}^{n\times r}\times\mathbb{R}^{n+1}\ldots\times\mathbb{R}^{n+1}$ to the tangent space $T_{y}\mathcal{M}$. Exploiting the product and Riemannian structure of $\mathcal{M}$, the projection operator characterization is obtained as the Cartesian product of the individual tangent space projection operator on the Stiefel and hyperbolic manifolds, both of which are well known. Specifically, if $(\zeta_{{\mathbf{U}}},\zeta_{\bar{{\mathbf{z}}}_{i}},\ldots,\zeta_{\bar{{\mathbf{z}}}_{m}})\in\mathbb{R}^{n\times r}\times\mathbb{R}^{n+1}\ldots\times\mathbb{R}^{n+1}$, then its projection onto the tangent space $T_{y}\mathcal{M}$ is given by

[TABLE]

4.2 Retraction

An optimization algorithm on manifold requires computation of search direction and then following along it. While the computation of the search direction follows from the notions in Section 4.1, in this section we develop the notion of “moving” along a search direction on the manifold. This is characterized by the retraction operation, which is the generalization of the the exponential map (that follows the geodesic) on the manifold. The retraction operator $R_{y}$ takes in a tangent vector at $T_{y}\mathcal{M}$ and outputs an element on the manifold by approximating the geodesic [23, Definition 4.1.1].

Exploiting the product space of $\mathcal{M}$, a natural expression of the retraction operator is obtained by the Cartesian product of the individual retraction operations on the Stiefel and hyperbolic manifolds. If $\xi_{y}\in T_{y}\mathcal{M}$, then the retraction operation is given by

[TABLE]

where $\xi_{y}=(\xi_{{\mathbf{U}}},\xi_{\bar{{\mathbf{z}}}_{i}},\ldots,\xi_{\bar{{\mathbf{z}}}_{m}})$, $\|\xi_{{\bar{{\mathbf{z}}}}_{i}}\|_{\mathcal{L}}=\sqrt{\left\langle\xi_{{\bar{{\mathbf{z}}}}_{i}},\xi_{\bar{{\mathbf{z}}}_{i}}\right\rangle_{\mathcal{L}}}$ for all $i$, $\mathrm{uf}(\cdot)$ extracts the orthogonal factor of a matrix, i.e., $\mathrm{uf}(\mathbf{A})=\mathbf{A}(\mathbf{A}^{\top}\mathbf{A})^{-1/2}$ .

4.3 Riemannian gradient and Hessian computations

Finally, we require the expressions of the Riemannian gradient and Hessian of $f$ on $\mathcal{M}$. To this end, we first compute the derivatives of $f$ in the Euclidean space. Let $\nabla_{y}f$ be the first derivative of $f$ and its Euclidean directional derivative along $\xi_{y}$ is $\mathrm{D}\nabla_{y}f[\xi_{y}]$. The expressions of the partial derivatives for the squared Euclidean distance based loss functions mentioned in Section 3 are straightforward to compute. When the loss function is based on the squared hyperbolic distance (3), the expressions for $\nabla_{y}f$ and $\mathrm{D}\nabla_{y}[\xi_{y}]$ are discussed in [24].

Once the partial derivatives of $f$ are known, converting them to their Riemannian counterparts on $\mathcal{M}$ follows from the theory of Riemannian optimization [23, Chapter 3]. The expressions are

[TABLE]

where $\Pi_{y}$ is the orthogonal projection operator (8).

4.4 Riemannian trust-region algorithm

The Riemannian trust-region (TR) algorithm approximates the function $f$ with a second-order model at every iteration. The second-order model (which is called the trust-region sub-problem) makes use of the Riemannian gradient and Hessian computations as shown in Section 4.3. The trust-region sub-problem is then solved efficiently (using an iterative quadratic optimization solver, e.g., with the truncated conjugate gradient algorithm) to obtain a candidate search direction. If the candidate search leads to an appreciable decrease in the function $f$, then it is accepted else it is rejected [23, Chapter 7]. Algorithm 1 summarizes the key steps of the proposed trust-region algorithm for solving (5).

4.5 Computational complexity

The manifold-related ingredients cost $O(nr^{2}+mr)$. For example, the computation of the Riemannian gradient in (10) involves only the tangent space projection operation that costs $O(nr^{2}+mr)$. Similarly, the retraction operation costs $O(nr^{2}+mr+r^{3})$.

The computations of $f$ and its derivatives cost $O(nmr)$ (for all the three choices of the loss function in Section 3). The overall computational cost per iteration of our implementation is, therefore, $O(nmr)$.

4.6 Numerical implementation

We use the Matlab toolbox Manopt [25] to implement Algorithm 1 for (5). Manopt comes with a well-implemented generic Riemannian trust-region solver, which can be used appropriately to solve (5) by providing the necessary optimization-related ingredients mentioned earlier. The Matlab codes are available at https://pratikjawanpuria.com.

5 Experiments

In this section, we evaluate the performance of the proposed low-rank parameterization of hyperbolic embeddings. In particular, we compare the quality of the low-rank hyperbolic embeddings obtained by minimizing the three different loss functions discussed in Section 3.

Experimental setup and evaluation metric

We are provided with the hyperbolic embeddings corresponding to a hierarchical entity such as nodes of a tree or a graph. We also have the ground truth information of the given tree (or graph). Let $(\mathcal{G},\mathcal{E})$ represents the ground truth, where $\mathcal{G}$ is the set of nodes and $\mathcal{E}=\{(u,v)\}$ be the set of edges between the nodes ($u,v\in\mathcal{G}$). Hyperbolic embeddings can be employed to reconstruct the ground truth since a low hyperbolic distance (3) between a pair of nodes implies a high probability of an edge between them. However, such a reconstruction may also incorporate errors such as missing out on an edge or adding a non-existent edge.

We measure the quality of the hyperbolic embeddings as follows: let $u$ and $v$ be a pair of nodes in $\mathcal{G}$ such that $(u,v)\in\mathcal{E}$. Let $\bar{\mathbf{u}}$ and $\bar{\mathbf{v}}$ be the hyperbolic embeddings corresponding to $u$ and $v$, respectively. We compute the hyperbolic distance $d_{\mathbb{H}}(\bar{\mathbf{u}},\bar{\mathbf{v})}$ (3) and rank it among the distance corresponding to all untrue edges from $u$, i.e., $\{d_{\mathbb{H}}(u,w)|(u,w)\notin\mathcal{E}\}$. We then compute the mean average precision (MAP) of the ranking. The MAP score is a commonly employed metric for evaluating graph embeddings [3, 13, 26, 27]. Overall, we compare the quality of the proposed low-rank approximation by comparing the MAP score of the original high dimensional embeddings and the low-rank embeddings.

We obtain the original hyperbolic embeddings from the implementation provided by [3]. It should be noted that [3] learns the hyperbolic embeddings from the Poincaré model and we employ the transformation discussed in Section 2 to obtain embeddings corresponding to the hyperboloid model. It should be mentioned that though [13] directly learns hyperbolic embeddings from the hyperboloid model, its implementation is not available.

Datasets

We perform experiments on the mammal and noun subtrees of the WordNet database[20]. WordNet is a lexical database and among other things, it also provides relations between pairs of concepts.

The ‘mammal’ dataset has mammal as the root node, with ‘is-a’ (hypernymy) relationship defining the edges. As an example, it has relationships such as ‘rodent’ is-a ‘mammal’, ‘squirrel’ is-a ‘rodent’, etc. Hence, there exists an edge from the ‘mammal’ node to ‘rodent’ node and from ‘rodent’ node to ‘squirrel’ node. The WordNet mammal subtree consists of $|\mathcal{G}|=1\,180$ nodes and $|\mathcal{E}|=6\,540$ edges. A part of this subtree is displayed in Figure 1(a).

Similarly, the ‘noun’ dataset is also a subtree of WordNet database. Examples in this subtree include ‘photograph’ is-a ‘object’, ’bronchitis’ is-a ‘disease’, ‘disease’ is-a ‘entity’, etc. It consists of $|\mathcal{G}|=82\,115$ nodes and $|\mathcal{E}|=743\,086$ edges.

Results

We compare the performance of the proposed low-rank approximation of hyperbolic embeddings with the three loss functions discussed in Section 3. Table 1 reports the results on the mammal dataset with different values of rank $r=\{5,10,20,50,100,200,300\}$. The original $300$-dimensional hyperbolic embeddings for the mammal subtree achieve a MAP score of $0.9501$. We observe that all the three methods are able to obtain MAP scores very close to the original embeddings with rank $r\geq 50$. In addition, Method-1 and Method-2 perform well even in very low-rank setting ($r=5$). This hints that penalizing with the Euclidean distance may be more suitable than compared to the hyperbolic distance (3) for approximating hyperbolic embeddings when the given rank is very small.

The results on the noun dataset are reported in Table 2. This dataset is challenging because of its scale and relatively low reconstruction performance of the original hyperbolic embeddings. The original $100$-dimensional hyperbolic embeddings for the noun subtree achieve a MAP score of $0.8070$. We observe that at rank $r=20$ our methods are able to get within $90\%$ of the performance obtained by the original embeddings.

6 Conclusion and Future work

Recently, hyperbolic embeddings have gained popularity in many machine learning applications because of their ability to model complex networks. In this paper, we have looked at scenarios where hyperbolic embeddings are potentially high dimensional and how to compress them using a low-rank factorization model. While low-rank decomposition of Euclidean embeddings are well-known, that of hyperbolic embeddings has not been well-studied. To this end, we have proposed a systematic approach to compute low-rank approximations of hyperbolic embeddings. Our approach allows to decompose a high dimensional hyperbolic embedding ($\bar{{\mathbf{x}}}$) into a product of low-dimensional subspace (${\mathbf{U}}$) and a smaller dimensional hyperbolic embedding ($\bar{{\mathbf{z}}}$).

We modeled the learning problem as an optimization problem on manifolds. Various optimization-related notions were presented to implement a Riemannian trust-region algorithm. Our experiments showed the benefit of the proposed low-rank approximations on real-world datasets.

As a future research direction, we would like to explore how low-rank hyperbolic embeddings are useful in downstream applications. Another research direction could be on developing methods to compute a “good” rank of hyperbolic embeddings.