NetSMF: Large-Scale Network Embedding as Sparse Matrix Factorization

TL;DR

NetSMF introduces a scalable large-scale network embedding method by sparsifying the matrix involved in spectral factorization, achieving high efficiency and effectiveness for massive networks.

Contribution

The paper proposes NetSMF, a novel sparse matrix factorization approach for large-scale network embedding, improving scalability and maintaining embedding quality.

Findings

NetSMF is the only method achieving high efficiency and effectiveness among benchmarks.

It can generate embeddings for networks with tens of millions of nodes in 24 hours.

Compared to DeepWalk, NetSMF is significantly faster, reducing computation from months to a day.

Abstract

We study the problem of large-scale network embedding, which aims to learn latent representations for network mining applications. Previous research shows that 1) popular network embedding benchmarks, such as DeepWalk, are in essence implicitly factorizing a matrix with a closed form, and 2)the explicit factorization of such matrix generates more powerful embeddings than existing methods. However, directly constructing and factorizing this matrix---which is dense---is prohibitively expensive in terms of both time and space, making it not scalable for large networks. In this work, we present the algorithm of large-scale network embedding as sparse matrix factorization (NetSMF). NetSMF leverages theories from spectral sparsification to efficiently sparsify the aforementioned dense matrix, enabling significantly improved efficiency in embedding learning. The sparsified matrix is spectrally close to the original dense one with a theoretically bounded approximation error, which helps maintain the representation power of the learned embeddings. We conduct experiments on networks of various scales and types. Results show that among both popular benchmarks and factorization based methods, NetSMF is the only method that achieves both high efficiency and effectiveness. We show that NetSMF requires only 24 hours to generate effective embeddings for a large-scale academic collaboration network with tens of millions of nodes, while it would cost DeepWalk months and is computationally infeasible for the dense matrix factorization solution. The source code of NetSMF is publicly available (https://github.com/xptree/NetSMF).

Tables

Table 1. Table 1. The comparison between NetSMF and other popular network embedding algorithms.

	LINE	DeepWalk	node2vec	NetMF	NetSMF
Efficiency	$\surd$				$\surd$
Global context		$\surd$	$\surd$	$\surd$	$\surd$
Theoretical guarantee				$\surd$	$\surd$
High-order proximity			$\surd$

Table 2. Table 2. Notations.

Notation	Description
$G$	input network
$V$	vertex set of $G$ with $\left|V\right|$=n
$E$	edge set of $G$ with $\left|E\right|=m$
$𝑨$	adjacency matrix of $G$
$𝑫$	degree matrix of $G$
$\mathrm{vol}\left(G\right)$	volume of $G$
$b$	number of negative samples
$T$	context window size
$d$	embedding dimension
$𝑳$	random-walk molynomial of $G$ (Eq. (4))
$\widetilde{𝑳}$	$𝑳$’s sparsifier
$𝑴$	$\frac{1}{T}{\sum }_{r=1}^T{\left({𝑫}^{-1}𝑨\right)}^r{𝑫}^{-1}$
$\widetilde{𝑴}$	$𝑴$’s sparsifier
$\mathrm{trunc}\_{\log }^{\circ }\left(\frac{\mathrm{vol}\left(G\right)}{b}𝑴\right)$	NetMF matrix
$\mathrm{trunc}\_{\log }^{\circ }\left(\frac{\mathrm{vol}\left(G\right)}{b}\widetilde{𝑴}\right)$	NetMF matrix sparisifier
$M$	number of non-zeros in $\widetilde{𝑳}$
$ϵ$	approximation factor
$\left[x\right]$	set $\{1,2,\cdots ,x\}$ for positive integer $x$

Table 3. Table 3. Time and Space Complexity of NetSMF.

	Time	Space
Step 1	$O\left(MT\log n\right)$ for weighted networks $O\left(MT\right)$ for unweighted networks	$O\left(M+n+m\right)$
Step 2	$O\left(M\right)$	$O\left(M+n\right)$
Step 3	$O\left(Md+n{d}^2+d^3\right)$	$O\left(M+nd\right)$

Table 4. Table 4. Statistics of Datasets.

Dataset	BlogCatalog	PPI	Flickr	YouTube	OAG
$\left|V\right|$	10,312	3,890	80,513	1,138,499	67,768,244
$\left|E\right|$	333,983	76,584	5,899,882	2,990,443	895,368,962
#labels	39	50	195	47	19

Table 5. Table 5. Efficiency comparison. The running time includes filesystem IO and computation time. “–” indicates that the corresponding algorithm fails to complete within one week. “ × \times ” indicates that the corresponding algorithm is unable to handle the computation due to excessive space and memory consumption.

	LINE	DeepWalk	node2vec	NetMF	NetSMF
BlogCatalog	40 mins	12 mins	56 mins	2 mins	13 mins
PPI	41 mins	4 mins	4 mins	16 secs	10 secs
Flickr	42 mins	2.2 hours	21 hours	2 hours	48 mins
YouTube	46 mins	1 day	4 days	$\times$	4.1 hours
OAG	2.6 hours	–	–	$\times$	24 hours

Equations

$${\log }^{\circ }\left(\frac{\mathrm{vol}(G)}{b}\mathbf{M}\right),$$

$$\mathbf{M}=\frac{1}{T}\sum _{r=1}^T({\mathbf{D}}^{-1}\mathbf{A}{)}^r{\mathbf{D}}^{-1},$$

$$trunc\_lo{\mathrm{g}}^{\circ }\left(\frac{\mathrm{vol}(G)}{b}\mathbf{M}\right).$$

$$\forall \mathbf{x}\in {\mathbb{R}}^n,(1-ϵ)\cdot {\mathbf{x}}^{\top }\tilde{\mathbf{L}}\mathbf{x}\le {\mathbf{x}}^{\top }\mathbf{L}\mathbf{x}\le (1+ϵ)\cdot {\mathbf{x}}^{\top }\tilde{\mathbf{L}}\mathbf{x}.$$

$$\mathbf{L}=\mathbf{D}-\sum _{r=1}^T{\alpha }_r\mathbf{D}{\left({\mathbf{D}}^{-1}\mathbf{A}\right)}^r,$$

$$\mathbf{M}={\mathbf{D}}^{-1}\left(\mathbf{D}-\mathbf{L}\right){\mathbf{D}}^{-1}.$$

$$trunc\_lo{\mathrm{g}}^{\circ }\left(\frac{\mathrm{vol}(G)}{b}\tilde{\mathbf{M}}\right).$$

$$Z(\mathbf{p})=\sum _{i=1}^r\frac{2}{{\mathbf{A}}_{u_{i-1},u_i}},$$

$${\Vert trunc\_lo{\mathrm{g}}^{\circ }\left(\frac{\mathrm{vol}(G)}{b}\tilde{\mathbf{M}}\right)-trunc\_lo{\mathrm{g}}^{\circ }\left(\frac{\mathrm{vol}(G)}{b}\mathbf{M}\right)\Vert }_F\le \frac{4ϵ\mathrm{vol}(G)}{b\sqrt{d_{\min }}}\sqrt{\sum _{i=1}^n\frac{1}{d_i}}.$$

$${\lambda }_i=\underset{\dim (U)=i}{\min }\underset{\mathbf{x}\in U,{\Vert x\Vert }_2=1}{\max }{\mathbf{x}}^{\top }\mathbf{A}\mathbf{x}.$$

$${\sigma }_{i+j-1}(\mathbf{BC})\le {\sigma }_i(\mathbf{B})\times {\sigma }_j(\mathbf{C})$$

$$\mathcal{L}={\mathbf{D}}^{-1/2}\mathbf{L}{\mathbf{D}}^{-1/2}=\mathbf{I}-\sum _{r=1}^T{\alpha }_r{\left({\mathbf{D}}^{-1/2}\mathbf{A}{\mathbf{D}}^{-1/2}\right)}^r$$

$$\frac{1}{1+ϵ}{\mathbf{x}}^{\top }\mathbf{L}\mathbf{x}\le {\mathbf{x}}^{\top }\tilde{\mathbf{L}}\mathbf{x}\le \frac{1}{1-ϵ}{\mathbf{x}}^{\top }\mathbf{L}\mathbf{x}.$$

$$\begin{array}{rl}& \frac{1}{1+ϵ}{\mathbf{y}}^{\top }\mathcal{L}\mathbf{y}\le {\mathbf{y}}^{\top }\tilde{\mathcal{L}}\mathbf{y}\le \frac{1}{1-ϵ}{\mathbf{y}}^{\top }\mathcal{L}\mathbf{y}\\ ⟹& \left|{\mathbf{y}}^{\top }(\tilde{\mathcal{L}}-\mathcal{L})\mathbf{y}\right|\le \frac{ϵ}{1-ϵ}{\mathbf{y}}^{\top }\mathcal{L}\mathbf{y}<2ϵ{\mathbf{y}}^{\top }\mathcal{L}\mathbf{y}.\end{array}$$

$$\left|{\lambda }_i(\tilde{\mathcal{L}}-\mathcal{L})\right|\le 2ϵ{\lambda }_i(\mathcal{L})<4ϵ.$$

$$\begin{array}{rl}{\sigma }_i\left(\tilde{\mathbf{M}}-\mathbf{M}\right)& \le {\sigma }_i\left({\mathbf{D}}^{-1/2}\right)\times {\sigma }_1\left(\tilde{\mathcal{L}}-\mathcal{L}\right)\times {\sigma }_1\left({\mathbf{D}}^{-1/2}\right)\\ & \le \frac{1}{\sqrt{d_i}}\times 4ϵ\times \frac{1}{\sqrt{d_{\min }}}=\frac{4ϵ}{\sqrt{d_i{d}_{\min }}}.\end{array}$$

$$\begin{array}{rl}& {\Vert trunc\_lo{\mathrm{g}}^{\circ }\left(\frac{\mathrm{vol}(G)}{b}\tilde{\mathbf{M}}\right)-trunc\_lo{\mathrm{g}}^{\circ }\left(\frac{\mathrm{vol}(G)}{b}\mathbf{M}\right)\Vert }_F\\ \le & {\Vert \frac{\mathrm{vol}(G)}{b}\tilde{\mathbf{M}}-\frac{\mathrm{vol}(G)}{b}\mathbf{M}\Vert }_F=\frac{\mathrm{vol}(G)}{b}{\Vert \tilde{\mathbf{M}}-\mathbf{M}\Vert }_F\\ =& \frac{\mathrm{vol}(G)}{b}\sqrt{\sum _{i\in [n]}{\sigma }_i^2(\tilde{\mathbf{M}}-\mathbf{M})}\le \frac{4ϵ\mathrm{vol}(G)}{b\sqrt{d_{\min }}}\sqrt{\sum _{i=1}^n\frac{1}{d_i}}.\end{array}$$

$$w(\mathbf{p})=\frac{{\prod }_{i=1}^r{\mathbf{A}}_{u_{i-1},u_i}}{{\prod }_{i=1}^{r-1}{\mathbf{D}}_{u_i}}.$$

$$\frac{w(\mathbf{p})}{\tau (\mathbf{p})M}=\frac{w(\mathbf{p})}{\frac{w(\mathbf{p})Z(\mathbf{p})}{2rm}\times M}=\frac{2rm}{MZ(\mathbf{p})}.$$

Full text

In this work, we present the algorithm of large-scale network embedding as sparse matrix factorization (NetSMF). NetSMF leverages theories from spectral sparsification to efficiently sparsify the aforementioned dense matrix, enabling significantly improved efficiency in embedding learning. The sparsified matrix is spectrally close to the original dense one with a theoretically bounded approximation error, which helps maintain the representation power of the learned embeddings. We conduct experiments on networks of various scales and types. Results show that among both popular benchmarks and factorization based methods, NetSMF is the only method that achieves both high efficiency and effectiveness. We show that NetSMF requires only 24 hours to generate effective embeddings for a large-scale academic collaboration network with tens of millions of nodes, while it would cost DeepWalk months and is computationally infeasible for the dense matrix factorization solution. The source code of NetSMF is publicly available111https://github.com/xptree/NetSMF.

††journalyear: 2019††copyright: iw3c2w3††conference: Proceedings of the 2019 World Wide Web Conference; May 13–17, 2019; San Francisco, CA, USA††booktitle: Proceedings of the 2019 World Wide Web Conference (WWW ’19), May 13–17, 2019, San Francisco, CA, USA††doi: 10.1145/3308558.3313446††isbn: 978-1-4503-6674-8/19/05

1. Introduction

Recent years have witnessed the emergence of network embedding, which offers a revolutionary paradigm for modeling graphs and networks (Hamilton et al., 2017). The goal of network embedding is to automatically learn latent representations for objects in networks, such as vertices and edges. Significant lines of research have shown that the latent representations are capable of capturing the structural properties of networks, facilitating various downstream network applications, such as vertex classification and link prediction (Perozzi et al., 2014; Tang et al., 2015; Grover and Leskovec, 2016; Dong et al., 2017).

Over the course of its development, the DeepWalk (Perozzi et al., 2014), LINE (Tang et al., 2015), and node2vec (Grover and Leskovec, 2016) models have been commonly considered as powerful benchmark solutions for evaluating network embedding research. The advantage of LINE lies in its scalability for large-scale networks as it only models the first- and second-order proximities. That is to say, its embeddings lose the multi-hop dependencies in networks. DeepWalk and node2vec, on the other hand, leverage random walks on graphs and skip-gram (Mikolov et al., 2013a) with large context sizes to model nodes further away (i.e., global structures). Consequently, it is computationally more expensive for DeepWalk and node2vec to handle large-scale networks. For example, with the default parameter settings (Perozzi et al., 2014), DeepWalk requires months to embed an academic collaboration network of 67 million vertices and 895 million edges222With the default DeepWalk parameters (walk length: 40 and $\#$walk per node: 80), 214+ billion nodes (67M$\times$40$\times$80) with a vocabulary size of 67 million are fed into skip-gram. As a reference, Mikolov et al. reported that training on Google News of 6 billion words and a vocabulary size of only 1 million cost 2.5 days with 125 CPU cores (Mikolov et al., 2013a).. The node2vec model, which performs high-order random walks, takes more time than DeepWalk to learn embeddings.

More recently, a study shows that both the DeepWalk and LINE methods can be viewed as implicit factorization of a closed-form matrix (Qiu et al., 2018). Building upon this theoretical foundation, the NetMF method was instead proposed to explicitly factorize this matrix, achieving more effective embeddings than DeepWalk and LINE. Unfortunately, it turns out that the matrix to be factorized is an $n\times n$ dense one with $n$ being the number of vertices in the network, making it prohibitively expensive to directly construct and factorize for large-scale networks.

In light of these limitations of existing methods (See the summary in Table 1), we propose to study representation learning for large-scale networks with the goal of achieving efficiency, capturing global structural contexts, and having theoretical guarantees. Our idea is to find a sparse matrix that is spectrally close to the dense NetMF matrix implicitly factorized by DeepWalk. The sparsified matrix requires a lower cost for both construction and factorization. Meanwhile, making it spectrally close to the original NetMF matrix can guarantee that the spectral information of the network is maintained, and the embeddings learned from the sparse matrix is as powerful as those learned from the dense NetMF matrix.

In this work, we present the solution to network embedding learning as sparse matrix factorization (NetSMF). NetSMF comprises three steps. First, it leverages the spectral graph sparsification technique (Cheng et al., 2015b, a) to find a sparsifier for a network’s random-walk matrix-polynomial. Second, it uses this sparsifier to construct a matrix with significantly fewer non-zeros than, but spectrally close to, the original NetMF matrix. Finally, it performs randomized singular value decomposition to efficiently factorize the sparsified NetSMF matrix, yielding the embeddings for the network.

With this design, NetSMF offers both efficiency and effectiveness with guarantees, as the approximation error of the sparsified matrix is theoretically bounded. We conduct experiments in five networks, which are representative of different scales and types. Experimental results show that for million-scale or larger networks, NetSMF achieves orders of magnitude speedup over NetMF, while maintaining competitive performance for the vertex classification task. In other words, both NetSMF and NetMF outperform well-recognized network embedding benchmarks (i.e., DeepWalk, LINE, and node2vec), but NetSMF addresses the computation challenge faced by NetMF.

To summarize, we introduce the idea of network embedding as sparse matrix factorization and present the NetSMF algorithm, which makes the following contributions to network embedding:

**Efficiency. ** NetSMF reaches significantly lower time and space complexity than NetMF. Remarkably, NetSMF is able to generate embeddings for a large-scale academic network of 67 million vertices and 895 million edges on a single server in 24 hours, while it would cost months for DeepWalk and node2vec, and is computationally infeasible for NetMF on the same hardware.

**Effectiveness. ** NetSMF is capable of learning embeddings that maintain the same representation power as the dense matrix factorization solution, making it consistently outperform DeepWalk and node2vec by up to 34% and LINE by up to 100% for the multi-label vertex classification task in networks.

**Theoretical Guarantee. ** NetSMF’s efficiency and effectiveness are theoretically backed up. The sparse NetSMF matrix is spectrally close to the exact NetMF matrix, and the approximation error can be bounded, maintaining the representation power of its sparsely learned embeddings.

2. Preliminaries

Commonly, the problem of network embedding is formalized as follows: Given an undirected and weighted network $G=(V,E,\bm{A})$ with $V$ as the vertex set of $n$ vertices, $E$ as the edge set of $m$ edges, and $\bm{A}$ as the adjacency matrix, the goal is to learn a function $V\rightarrow\mathbb{R}^{d}$ that maps each vertex to a $d$-dimensional ($d\ll n$) vector that captures its structural properties, e.g., community structures. The vector representation of each vertex can be fed into downstream applications such as link prediction and vertex classification.

One of the pioneering work on network embedding is the DeepWalk model  (Perozzi et al., 2014), which has been consistently considered as a powerful benchmark over the past years (Hamilton et al., 2017). In brief, DeepWalk is coupled with two steps. First, it generates several vertex sequences by random walks over a network; Second, it applies the skip-gram model (Mikolov et al., 2013b) on the generated vertex sequences to learn the latent representations for each vertex. Commonly, skip-gram is parameterized with the context window size $T$ and the number of negative samples $b$. Recently, a theoretical study (Qiu et al., 2018) reveals that DeepWalk essentially factorizes a matrix derived from the random walk process. More formally, it proves that when the length of random walks goes to infinity, DeepWalk implicitly and asymptotically factorizes the following matrix:

[TABLE]

where $\operatorname{vol}{(G)}=\sum_{i}\sum_{j}\bm{A}_{ij}$ denotes the volume of the graph, and

[TABLE]

where $\bm{D}=\operatorname{diag}{(d_{1},\cdots,d_{n})}$ is the degree matrix with $d_{i}=\sum_{j}\bm{A}_{ij}$ as the generalized degree of the $i$-th vertex. Note that $\operatorname{\log^{\circ}}(\bm{\cdot})$ represents the element-wise matrix logarithm (Horn and Johnson, 1991), which is different from the matrix logarithm. In other words, the matrix in Eq. (1) can be characterized as the result of applying element-wise matrix logarithm (i.e., $\operatorname{\log^{\circ}}$) to matrix $\frac{\operatorname{vol}{(G)}}{b}\bm{M}$.

The matrix in Eq. (1) offers an alternative view of the skip-gram based network embedding methods. Further, Qiu et al. provide an explicit matrix factorization approach named NetMF to learn the embeddings (Qiu et al., 2018). It shows that the accuracy for vertex classification based on the embeddings from NetMF outperforms that based on DeepWalk and LINE. Note that the matrix in Eq. (1) would be ill-defined if there exist a pair of vertices unreachable in $T$ hops, because $\log(0)=-\infty$. So following Levy and Goldberg (Levy and Goldberg, 2014), NetMF uses the logarithm truncated at point one, that is, $\operatorname{trunc\_log}(x)=\max(0,\log(x))$. Thus, NetMF targets to factorize the matrix

[TABLE]

In the rest of this work, we refer to the matrix in Eq. (3) as the NetMF matrix.

However, there exist a couple of challenges when leveraging the NetMF matrix in practice. First, almost every pair of vertices within distance $r\leq T$ correspond to a non-zero entry in the NetMF matrix. Recall that many social and information networks exhibit the small-world property where most vertices can be reached from each other in a small number of steps. For example, as of the year 2012, 92% of the reachable pairs in Facebook are at distance five or less (Backstrom et al., 2012). As a consequence, even if setting a moderate context window size (e.g., the default setting $T=10$ in DeepWalk), the NetMF matrix in Eq. (3) would be a dense matrix with $O(n^{2})$ number of non-zeros. The exact construction and factorization of such a matrix is impractical for large-scale networks. More concretely, computing the matrix power in Eq. (2) involves dense matrix multiplication which costs $O(n^{3})$ time; factorizing a $n\times n$ dense matrix is also time consuming. To reduce the construction cost, NetMF approximates $\bm{M}$ with its top eigen pairs. However, the approximated matrix is still dense, making this strategy unable to handle large networks.

In this work, we aim to address the efficiency and scalability limitation of NetMF, while maintaining its superiority in effectiveness. We list necessary notations and their descriptions in Table 2.

3. Network Embedding as Sparse Matrix Factorization (NetSMF)

In this section, we develop network embedding as sparse matrix factorization (NetSMF). We present the NetSMF method to construct and factorize a sparse matrix that approximates the dense NetMF matrix. The main technique we leverage is random-walk matrix-polynomial (molynomial) sparsification.

3.1. Random-Walk Molynomial Sparsification

We first introduce the definition of spectral similarity and the theorem of random-walk molynomial sparsification.

Definition 0.

(Spectral Similarity of Networks) Suppose $G=(V,E,\bm{A})$ and $\widetilde{G}=(V,\widetilde{E},\widetilde{\bm{A}})$ are two weighted undirected networks. Let $\bm{L}=\bm{D}_{G}-\bm{A}$ and $\widetilde{\bm{L}}=\bm{D}_{\widetilde{G}}-\widetilde{\bm{A}}$ be their Laplacian matrices, respectively. We define $G$ and $\widetilde{G}$ are $(1+\epsilon)$-spectrally similar if

[TABLE]

Theorem 2.

*(Spectral Sparsifiers of Random-Walk Molynomials (Cheng et al., 2015a, b))

For random-walk molynomial*

[TABLE]

where $\sum_{r=1}^{T}\alpha_{r}=1$ and $\alpha_{r}$ non-negative, one can construct, in time $O(T^{2}m\epsilon^{-2}\log^{2}n)$, a $(1+\epsilon)$-spectral sparsifier, $\widetilde{\bm{L}}$, with $O(n\log{n}\epsilon^{-2})$ non-zeros. For unweighted graphs, the time complexity can be reduced to $O(T^{2}m\epsilon^{-2}\log n)$.

To achieve a sparsifier $\widetilde{\bm{L}}$ with $O(\epsilon^{-2}n\log{n})$ non-zeros, the sparsification algorithm consists of two steps:

The first step obtains an initial sparsifier for $\bm{L}$ with $O(Tm\epsilon^{-2}\log{n})$ non-zeros. The second step then applies the standard spectral sparsification algorithm (Spielman and Srivastava, 2011) to further reduce the number of non-zeros to $O(\epsilon^{-2}n\log{n})$. In this work, we only adopt the first step because a sparsifier with $O(Tm\epsilon^{-2}\log{n})$ non-zeros is sparse enough for our task. Thus we skip the second step that involves additional computations. From now on, when referring to the random-walk molynomial sparsification algorithm in this work, we mean its first step only.

One can immediately observe that, if we set $\alpha_{r}=\frac{1}{T},r\in[T]$, the matrix $\bm{L}$ in Eq. (4) has a strong connection with the desired matrix $\bm{M}$ in Eq. (2). Formally, we have the following equation

[TABLE]

Thm. 2 can help us construct a sparsifier $\widetilde{\bm{L}}$ for matrix $\bm{L}$. Then we define $\widetilde{\bm{M}}\triangleq\bm{D}^{-1}(\bm{D}-\widetilde{\bm{L}})\bm{D}^{-1}$ by replacing $\bm{L}$ in Eq. (5) with its sparsifier $\widetilde{\bm{L}}$. One can observe that matrix $\widetilde{\bm{M}}$ is still a sparse one with the same order of magnitude of non-zeros as $\widetilde{\bm{L}}$. Consequently, instead of factorizing the dense NetMF matrix in Eq. (3), we can factorize its sparse alternative, i.e.,

[TABLE]

In the rest of this work, the matrix in Eq. (6) is referred to as the NetMF matrix sparsifier.

3.2. The NetSMF Algorithm

In this section, we formally describe the NetSMF algorithm, which consists of three steps: random-walk molynomial sparsification, NetMF sparsifier construction, and truncated singular value decomposition.

**Step 1: Random-Walk Molynomial Sparsification. ** To achieve the sparsifier $\widetilde{\bm{L}}$, we adopt the algorithm in Cheng et al. (2015b). The algorithm starts from creating a network $\widetilde{G}$ that has the same vertex set as $G$ and an empty edge set (Alg. 1, Line 1). Next, the algorithm constructs a sparsifier with $O(M)$ non-zeros by repeating the PathSampling algorithm for $M$ times. In each iteration, it picks an edge $e\in E$ and an integer $r\in[T]$ uniformly (Alg. 1, Line 3-4). Then, the algorithm uniformly draws an integer $k\in[r]$ and performs $(k-1)$-step and $(r-k)$-step random walks starting from the two endpoints of edge $e$ respectively (Alg. 2, Line 3-4). The above process samples a length-$r$ path $\bm{p}=(u_{0},u_{1},\cdots,u_{r})$. At the same time, the algorithm keeps track of $Z(\bm{p})$, which is defined by

[TABLE]

and then adds a new edge $(u_{0},u_{r})$ with weight $\frac{2rm}{MZ(\bm{p})}$ to $\widetilde{G}$ (Alg. 1, Line 6).333Details about how the edge weight is derived can be found in Thm. 7 in Appendix. Parallel edges in $\widetilde{G}$ will be merged into one single edge, with their weights summed up together. Finally, the algorithm computes the Laplacian of $\widetilde{G}$, which is the sparsifier $\widetilde{\bm{L}}$ as we desired (Alg. 1, Line 8). This step gives us a sparsifier with $O(M)$ non-zeros.

**Step 2: Construct a NetMF Matrix Sparsifier. ** As we have discussed at the end of Section 3.1, after constructing a sparsifier $\widetilde{\bm{L}}$, we can plug it into Eq. (5) to obtain a NetMF matrix sparsifier as shown in Eq. (6) (Alg. 1, Line 9-10). This step does not change the order of magnitude of non-zeros in the sparsifier.

**Step 3: Truncated Singular Value Decomposition. ** The final step is to perform truncated singular value decomposition (SVD) on the constructed NetMF matrix sparsifier (Eq. (6)). However, even the sparsifier only has $O(M)$ number of non-zeros, performing exact SVD is still time consuming. In this work, we leverage a modern randomized matrix approximation technique—Randomized SVD—developed by Halko et al. (2011). Due to space constraint, we cannot include many details. Briefly speaking, the algorithm projects the original matrix to a low-dimensional space through a Gaussian random matrix. One only needs to perform traditional SVD (e.g. Jacobi SVD) on a $d\times d$ small matrix. We list the pseudocode algorithm in Alg. 3. Another advantage of SVD is that we can determine the dimensionality of embeddings by using, for example, Cattell’s Scree test (Cattell, 1966). In the test, we plot the singular values and select a rank $d$ such that there is a clear drop in the magnitudes or the singular values start to even out. More details will be discussed in Section 4.

**Complexity Analysis. ** Now we analyze the time and space complexity of NetSMF, as summarized in Table 3. As for step 1, we call the PathSampling algorithm for $M$ times, during each of which it performs $O(T)$ steps of random walks over the network. For unweighted networks, sampling a neighbor requires $O(1)$ time, while for weighted networks, one can use roulette wheel selection to choose a neighbor in $O(\log{n})$. It taks $O(M)$ space to store $\widetilde{G}$, while the additional $O(n+m)$ space comes from the storage of the input network. As for step 2, it takes $O(M)$ time to perform the transformation in Eq. (5) and the element-wise truncated logarithm in Eq. (6). The additional $O(n)$ space is spent in storing the degree matrix. As for step 3, $O(Md)$ time is required to compute the product of a row-major sparse matrix and a dense matrix (Alg. 3, Lines 3 and 5); $O(nd^{2})$ time is spent in Gram-Schmidt orthogonalization (Alg. 3, Lines 4 and 8); $O(d^{3})$ time is spent in Jacobi SVD (Alg. 3, Line 10).

**Connection to NetMF. ** The major difference between NetMF and NetSMF lies in the approximation strategy of the NetMF matrix in Eq. (3). As we mentioned in Section 2, NetMF approximates it with a dense matrix, which brings new space and computation challenges. In this work, NetSMF aims to find a sparse approximator to the NetMF matrix by leveraging theories and techniques from spectral graph sparsification.

**Example. ** We provide a running example to help understand the NetSMF algorithm. Suppose we want to learn embeddings for a network with $n=10^{6}$ vertices, $m=10^{7}$ edges, context window size $T=10$, and approximation factor $\epsilon=0.1$. The NetSMF method calls the PathSampling algorithm for $M=Tm\epsilon^{-2}\log{n}\approx 1.4\times 10^{11}$ times and provides us with a NetMF matrix sparsifier with at most $1.4\times 10^{11}$ non-zeros (Notice that the reducer in Step 1 and $\operatorname{trunc\_log^{\circ}}$ in Step 2 will further sparsify the matrix, making $1.4\times 10^{11}$ an upper bound). The density of the sparsifier is at most $\frac{M}{n^{2}}\approx 14\%$. Then, when computing the sparse-dense matrix product in randomized SVD (Alg. 3, Lines 3 and 5), the sparseness of the factorized matrix can greatly accelerate the calculation. In comparison, NetMF must construct a dense matrix with $n^{2}=10^{12}$ non-zeros, which is an order of magnitude larger in terms of density. Also, the density of the sparsifier in NetSMF can be further reduced by using a larger $\epsilon$, while NetMF does not have this flexibility.

3.3. Approximation Error Analysis

In this section, we analyze the approximation error of the sparsification. We assume that we choose an approximation factor $\epsilon<0.5$. We first see how the constructed $\widetilde{\bm{M}}$ approximates $\bm{M}$ and then compare the NetMF matrix (Eq. (3)) against the NetMF matrix sparsifier (Eq. (6)). We use $\sigma_{i}$ to denote the $i$-th descending-order singular value of a matrix. We also assume the vertices’ degrees are sorted in ascending order, that is, $d_{\min}=d_{1}\leq d_{2}\cdots\leq d_{n}$.

{restatable}

theoremMerror

The singular value of $\widetilde{\bm{M}}-\bm{M}$ satisfies $\sigma_{i}(\widetilde{\bm{M}}-\bm{M})\leq\frac{4\epsilon}{\sqrt{d_{i}d_{\min}}},\forall i\in[n]$.

{restatable}

theoremlogerror

Let $\left\lVert\cdot\right\rVert_{F}$ be the matrix Frobenius norm. Then

[TABLE]

Proof.

See Appendix. ∎

**Discussion on the Approximation Error. ** The above bound is achieved without making assumptions about the input network. If we introduce some assumptions, say a bounded lowest degree $d_{\min}$ or a specific random graph model (e.g., Planted Partition Model or Extended Planted Partition Model), it is promising to explore tighter bounds by leveraging theorems in literature (Dasgupta et al., 2004; Chaudhuri et al., 2012).

3.4. Parallelization

Each step of NetSMF can be parallelized, enabling it to scale to very large networks. The parallelization design of NetSMF is introduced in Figure 1. Below we discuss the parallelization of each step in detail. At the first step, the paths in the PathSampling algorithm are sampled independently with each other. Thus we can launch multiple PathSampling workers simultaneously. Each worker handles a subset of the samples. Herein, we require that each worker is able to access the network data $G=(V,E,\bm{A})$ efficiently. There are many options to meet this requirement. The easiest one is to load a copy of the network data to each worker’s memory. When the network is extremely large (e.g., trillion scale) or workers have memory constraints, the graph engine should be designed to expose efficient graph query APIs to support graph operations such as random walks. At the end of this step, a reducer is designed to merge parallel edges and sum up their weights. If this step is implemented in a big data system such as Spark (Zaharia et al., 2010), the reduction step can be simply achieved by running a reduceByKey(+)444https://spark.apache.org/docs/latest/rdd-programming-guide.html function. After the reduction, the sparsifier $\widetilde{\bm{L}}$ is organized as a collection of triplets, a.k.a, COOrdinate format, with each indicating an entry of the sparsifier. The second step is the most straightforward step to scale up. When processing a triplet $(u,v,w)$, we can simply query the degree of vertices $u$ and $v$ and perform the transformation defined in Eq. (5) as well as the truncated logarithm in Eq. (6), which can be well parallelized. For the last step, we organize the sparsifier into row-major format. This format allows efficient multiplication between a sparse and a dense matrix (Alg. 3, Line 3 and 5). Other dense matrix operators (e.g., Gaussian random matrix generation, Gram-Schmidt orthogonalization and Jacobi SVD) can be easily accelerated by using multi-threading or common linear algebra libraries. In this work, we adopt a single-machine shared-memory implementation. We use OpenMP (Dagum and Menon, 1998) to parallelize NetSMF in our implementation555Code is publicly available at https://github.com/xptree/NetSMF.

4. Experiments

In this section, we evaluate the proposed NetSMF method on the multi-label vertex classification task, which has been commonly used to evaluate previous network embedding techniques (Perozzi et al., 2014; Tang et al., 2015; Grover and Leskovec, 2016; Qiu et al., 2018). We introduce our datasets and baselines in Section 4.1 and Section 4.2. We report experimental results and parameter analysis in Section 4.3 and Section 4.4, respectively.

4.1. Datasets

We employ five datasets for the prediction task, four of which are in relatively small scale but have been widely used in network embedding literature, including BlogCatalog, PPI, Flickr, and YouTube. The remaining one is a large-scale academic co-authorship network, which is at least two orders of magnitude larger than the largest one (YouTube) used in most network embedding studies. The statistics of these datasets are listed in Table 4.

**BlogCatalog (Tang and Liu, 2009a; Agarwal et al., 2009) ** is a network of social relationships of online bloggers. The vertex labels represent the interests of the bloggers.

**Protein-Protein Interactions (PPI) (Stark et al., 2010) ** is a subgraph of the PPI network for Homo Sapiens. The vertex labels are obtained from the hallmark gene sets and represent biological states.

**Flickr (Tang and Liu, 2009a) ** is the user contact network in Flickr. The labels represent the interest groups of the users.

**YouTube (Tang and Liu, 2009b) ** is a video-sharing website that allows users to upload, view, rate, share, add to their favorites, report, comment on videos. The users are labeled by the video genres they liked.

**Open Academic Graph (OAG)666www.openacademic.ai/oag/ ** is an academic graph indexed by Microsoft Academic (Sinha et al., 2015) and AMiner.org (Tang et al., 2008). We construct an undirected co-authorship network from OAG, which contains 67,768,244 authors and 895,368,962 collaboration edges. The vertex labels are defined to be the top-level fields of study of each author, such as computer science, physics and psychology. In total, there are 19 distinct fields (labels) and authors may publish in more than one field, making the associated vertices have multiple labels.

4.2. Baseline Methods

We compare NetSMF with NetMF (Qiu et al., 2018), LINE (Tang et al., 2015), DeepWalk (Perozzi et al., 2014), and node2vec (Grover and Leskovec, 2016). For NetSMF, NetMF, DeepWalk, and node2vec that allow multi-hop structural dependencies, the context window size $T$ is set to be 10, which is also the default setting used in both DeepWalk and node2vec. Across all datasets, we set the embedding dimension $d$ to be 128. We follow the common practice for the other hyper-parameter settings, which are introduced below.

**LINE. ** We use LINE with the second order proximity (i.e., LINE (2nd) (Tang et al., 2015)). We use the default setting of LINE’s hyper-parameters: the number of edge samples to be 10 billion and the negative sample size to be 5.

**DeepWalk. ** We present DeepWalk’s results with the authors’ preferred parameters, that is, walk length to be 40, the number of walks from each vertex to be 80, and the number of negative samples in skip-gram to be 5.

**node2vec. ** For the return parameter $p$ and in-out parameter $q$ in node2vec, we adopt the default setting that was used by its authors if available. Otherwise, we grid search $p,q\in\{0.25,0.5,1,2,4\}$. For a fair comparison, we use the same walk length and the number of walks per vertex as DeepWalk.

**NetMF. ** In NetMF, the hyper-parameter $h$ indicates the number of eigen pairs used to approximate the NetMF matrix. We choose $h=256$ for the BlogCatalog, PPI and Flickr datasets.

**NetSMF. ** In NetSMF, we set the number of samples $M=10^{3}\times T\times m$ for the PPI, Flickr, and YouTube datasets, $M=10^{4}\times T\times m$ for BlogCatalog, and $M=10\times T\times m$ for OAG in order to achieve desired performance. For both NetMF and NetSMF, we have $b=1$.

**Prediction Setting. ** We follow the same experiment and evaluation procedures that were performed in DeepWalk (Perozzi et al., 2014). First, we randomly sample a portion of labeled vertices for training and use the remaining for testing. For the BlogCatalog and PPI datasets, the training ratio varies from 10% to 90%. For Flickr, YouTube and OAG, the training ratio varies from 1% to 10%. We use the one-vs-rest logistic regression model implemented by LIBLINEAR (Fan et al., 2008) for the multi-label vertex classification task. In the test phase, the one-vs-rest model yields a ranking of labels rather than an exact label assignment. To avoid the thresholding effect, we take the assumption that was made in DeepWalk, LINE, and node2vec, that is, the number of labels for vertices in the test data is given (Perozzi et al., 2014; Tang et al., 2009; Grover and Leskovec, 2016). We repeat the prediction procedure ten times and evaluate the average performance in terms of both Micro-F1 and Macro-F1 scores (Tsoumakas et al., 2009). All the experiments are performed on a server with Intel Xeon E7-8890 CPU (64 cores), 1.7TB memory, and 2TB SSD hard drive.

4.3. Experimental Results

We summarize the prediction performance in Figure 2. To compare the efficiency of different algorithms, we also list the running time of each algorithm across all datasets, if available, in Table 5.

**NetSMF vs. NetMF. ** We first focus on the comparison between NetSMF and NetMF, since the goal of NetSMF is to address the efficiency and scalability issues of NetMF while maintaining its superiority in effectiveness. From Table 5, we observe that for YouTube and OAG, both of which contain more than one million vertices, NetMF fails to complete because of the excessive space and memory consumption, while NetSMF is able to finish in four hours and one day, respectively. For the moderate-size network Flickr, both methods are able to complete within one week, though NetSMF is 2.5$\times$ faster (i.e., 48 mins vs. 2 hours). For small-scale networks, NetMF is faster than NetSMF in BlogCatalog and is comparable to NetSMF in PPI in terms of running time. This is because when the input networks contain only thousands of vertices, the advantage of sparse matrix construction and factorization over its dense alternative could be marginalized by other components of the workflow.

In terms of prediction performance, Figure 2 suggests NetSMF and NetMF yield consistently the best results among all compared methods, empirically demonstrating the power of the matrix factorization framework for network embedding. In BlogCatalog, NetSMF has slightly worse performance than NetMF (on average less than 3.1% worse regarding both Micro- and Macro-F1). In PPI, the two leading methods’ performance are relatively indistinguishable in terms of both metrics. In Flickr, NetSMF achieves significantly better Macro-F1 than NetMF (by 3.6% on average), and also higher Micro-F1 (by 5.3% on average). Recall that NetMF uses a dense approximation of the matrix to factorize. These results show that the sparse spectral approximation used by NetSMF does not necessarily yield worse performance than the dense approximation used by NetMF.

*Overall, not only NetSMF improves the scalability, and the running time of NetMF by orders of magnitude for large-scale networks, it also has competitive, and sometimes better, performance. This demonstrates the effectiveness of our spectral sparsification based approximation algorithm. *

**NetSMF vs. DeepWalk, LINE $\&$ node2vec. ** We also compare NetSMF against common graph embedding benchmarks—DeepWalk, LINE, and node2vec. For the OAG dataset, DeepWalk and node2vec fail to finish the computation within one week, while NetSMF requires only 24 hours. Based on the publicly reported running time of skip-gram (Mikolov et al., 2013a), we estimate that DeepWalk and node2vec may require months to generate embeddings for the OAG dataset. In BlogCatalog, DeepWalk and NetSMF require similar computing time, while in Flickr, YouTube, and PPI, NetSMF is 2.75$\times$, 5.9$\times$, and 24$\times$ faster than DeepWalk, respectively. In all the datasets, NetSMF achieves 4–24$\times$ speedup over node2vec.

Moreover, the performance of NetSMF is significantly better than DeepWalk in BlogCatalog, PPI, and Flickr, by 7–34% in terms of Micro-F1 and 5–25% in terms of Macro-F1. In YouTube, NetSMF achieves comparable results to DeepWalk. Compared with node2vec, NetSMF achieves comparable performance in BlogCatalog and YouTube, and significantly better performance in PPI and Flickr. In summary, NetSMF consistently outperforms DeepWalk and node2vec in terms of both efficiency and effectiveness.

LINE has the best efficiency among all the five methods and together with NetSMF, they are the only methods that can generate embeddings for OAG within one week (and both finish in one day). However, it also has the worst prediction performance and consistently loses to others by a large margin across all datasets. For example, NetSMF beats LINE by 21% and 39% in Flickr, and by 30% and 100% in OAG in terms of Micro-F1 and Macro-F1, respectively.

In summary, LINE achieves efficiency at the cost of ignoring multi-hop dependencies in networks, which are supported by all the other four methods—DeepWalk, node2vec, NetMF, and NetSMF, demonstrating the importance of multi-hop dependencies for learning network representations.

*More importantly, among these four methods, DeepWalk achieves neither efficiency nor effectiveness superiority; node2vec achieves relatively good performance at the cost of efficiency; NetMF achieves effectiveness at the expense of significantly increased time and space costs; NetSMF is the only method that achieves both high efficiency and effectiveness, empowering it to learn effective embeddings for billion-scale networks (e.g., the OAG network with 0.9 billion edges) in one day on one modern server. *

4.4. Parameter Analysis

In this section, we discuss how the hyper-parameters influence the performance and efficiency of NetSMF. We report all the parameter analyses on the Flickr dataset with training ratio set to be 10%.

**How to Set the Embedding Dimension $d$. ** As mentioned in Section 3.1, SVD allows us to determine a “good” embedding dimension without supervised information. There are many methods available such as captured energy and Cattell’s Scree test (Cattell, 1966). Here we propose to use Cattell’s Scree test. Cattell’s Scree test plots the singular values and selects a rank $d$ such that there is a clear drop in the magnitudes or the singular values start to even out. In Flickr, if we sort the singular values in decreasing order, we can observe that the singular values approach 0 when the rank increases to around 100, as shown in Figure 3. In our experiments, by varying $d$ form $2^{4}$ to $2^{8}$, we reach the best performance at $d=128$, as shown in Figure 3, demonstrating the ability of our matrix factorization based NetSMF for automatically determining the embedding dimension.

**The Number of Non-Zeros $M$. ** In theory, $M$ = $O(Tm\epsilon^{-2}\log{n})$ is required to guarantee the approximation error (See Section 3.1). Without loss of generality, we empirically set $M$ to be $k\times T\times m$ where $k$ is chosen from 1, 10, 100, 200, 500, 1000, 2000 and investigate how the number of non-zeros influence the quality of learned embeddings. As shown in Figure 3, when increasing the number of non-zeros, NetSMF tends to have better prediction performance because the original matrix is being approximated more accurately. On the other hand, although increasing $M$ has a positive effect on the prediction performance, its marginal benefit diminishes gradually. One can observe that setting $M=1000\times T\times m$ (the second-to-the-right data point on each line in Figure 3) is a good choice that balances NetSMF’s efficiency and effectiveness.

**The Number of Threads. ** In this work, we use a single-machine shared memory implementation with multi-threading acceleration. We report the running time of NetSMF when setting the number of threads to be 1, 10, 20, 30, 60, respectively. As shown in Figure 3, NetSMF takes 12 hours to embed the Flickr network with one thread and 48 minutes to run with 30 threads, achieving a 15$\times$ speedup ratio (with ideal being 30$\times$). This relatively good sub-linear speedup supports NetSMF to scale up to very large-scale networks.

5. Related Work

In this section, we review the related work of network embedding, large-scale embedding algorithms, and spectral graph sparsification.

5.1. Network Embedding

Network embedding has been extensively studied over the past years (Hamilton et al., 2017). The success of network embedding has driven a lot of downstream network applications, such as recommendation systems (Ying et al., 2018). Briefly, recent work about network embedding can be categorized into three genres: (1) Skip-gram based methods that are inspired by word2vec (Mikolov et al., 2013a), such as LINE (Tang et al., 2015), DeepWalk (Perozzi et al., 2014), node2vec (Grover and Leskovec, 2016), metapath2vec (Dong et al., 2017), and VERSE (Tsitsulin et al., 2018); (2) Deep learning based methods such as (Ying et al., 2018; Kipf and Welling, 2017); (3) Matrix factorization based methods such as GraRep (Cao et al., 2015) and NetMF (Qiu et al., 2018). Among them, NetMF bridges the first and the third categories by unifying a collection of skip-gram based network embedding methods into a matrix factorization framework. In this work, we leverage the merit of NetMF and address its limitation in efficiency. Among literature, PinSage is notably a network embedding framework for billion-scale networks (Ying et al., 2018). The difference between NetSMF and PinSage lies in the following aspect. The goal of NetSMF is to pre-train general network embeddings in an unsupervised manner, while PinSage is a supervised graph convolutional method with both the objective of recommender systems and existing node features incorporated. That being said, the embeddings learned by NetSMF can be consumed by PinSage for downstream network applications.

5.2. Large-Scale Embedding Learning

Studies have attempted to optimize embedding algorithms for large datasets from different perspectives. Some focus on improving skip-gram model, while others consider it as matrix factorization.

**Distributed Skip-Gram Model. ** Inspired by word2vec (Mikolov et al., 2013b), most of the modern embedding learning algorithms are based on the skip-gram model. There is a sequence of work trying to accelerate the skip-gram model in a distributed system. For example, Ji et al. (2016) replicate the embedding matrix on multiple workers and synchronize them periodically; Ordentlich et al. (2016) distribute the columns (dimensions) of the embedding matrix to multiple executors and synchronize them with a parameter server (Li et al., 2014). Negative sampling is a key step in skip-gram, which requires to draw samples from a noisy distribution. Stergiou et al. (2017) focus on the optimization of negative sampling by replacing the roulette wheel selection with a hierarchical sampling algorithm based on the alias method. More recently, Wang et al. (2018) propose a billion-scale network embedding framework by heuristically partitioning the input graph to small subgraphs, and processing them separately in parallel. However, the performance of their framework highly relies on the quality of graph partition. The drawback for partition-based embedding learning is that the embeddings learned in different subgraphs do not share the same latent space, making it impossible to compare nodes across subgraphs.

**Efficient Matrix Factorization. ** Factorizing the NetMF matrix, either implicitly (e.g., LINE (Tang et al., 2015) and DeepWalk (Perozzi et al., 2014)) or explicitly (e.g., NetMF (Qiu et al., 2018)), encounters two issues. First, the denseness of this matrix makes computation expensive even for a moderate context window size (e.g., $T=10$). Second, the non-linear transformation, i.e., element-wise matrix logarithm, is hard to approximate. LINE (Tang et al., 2015) solves this problem by setting $T=1$. With such simplification, it achieves good scalability at the cost of prediction performance. NetSMF addresses these issues by efficiently sparsifying the dense NetMF matrix with a theoretically-bounded approximation error.

5.3. Spectral Graph Sparsification

Spectral graph sparsification has been studied for decades in graph theory (Teng et al., 2016). The task of graph sparsification is to approximate a “dense” graph by a “sparse” one that can be effectively used in place of the dense one (Teng et al., 2016), which arises in many applications such as scientific computing (Higham and Lin, 2011), machine learning (Cheng et al., 2015a; Calandriello et al., 2018) and data mining (Zhao, 2015). Our NetSMF model is the first work that incorporates spectral sparsification algorithms (Cheng et al., 2015b, a) into network embedding, which offers a powerful and efficient way to approximate and analyze the random-walk matrix-polynomial in the NetMF matrix.

6. Conclusion

In this work, we study network embedding with the goal of achieving both efficiency and effectiveness. To address the scalability challenges faced by the NetMF model, we propose to study large-scale network embedding as sparse matrix factorization. We present the NetSMF algorithm, which achieves a sparsification of the (dense) NetMF matrix. Both the construction and factorization of the sparsified matrix are fast enough to support very large-scale network embedding learning. For example, it empowers NetSMF to efficiently embed the Open Academic Graph in 24 hours, whose size is computationally intractable for the dense matrix factorization solution (NetMF). Theoretically, the sparsified matrix is spectrally close to the original NetMF matrix with an approximation bound. Empirically, our extensive experimental results show that the sparsely learned embeddings by NetSMF are as effective as those from the factorization of the NetMF matrix, leaving it outperform the common network embedding benchmarks—DeepWalk, LINE, and node2vec. In other words, among both matrix factorization based methods (NetMF and NetSMF) and common skip-gram based benchmarks (DeepWalk, LINE, and node2vec), NetSMF is the only model that achieves both efficiency and performance superiority.

**Future Work. ** NetSMF brings an efficient, effective, and guaranteed solution to network embedding learning. There are multiple tangible research fronts we can pursue. First, our current single-machine implementation limits the number of samples we can take for large networks. We plan to develop a multi-machine solution in the future to further scale NetSMF. Second, building upon NetSMF, we would like to efficiently and accurately learn embeddings for large-scale directed (Cohen et al., 2016), dynamic (Kapralov et al., 2017), and/or heterogeneous networks. Third, as the advantage of matrix factorization methods demonstrated, we are also interested in exploring the other matrix definitions that may be effective in capturing different structural properties in networks. Last, it would be also interesting to bridge matrix factorization based network embedding methods with graph convolutional networks.

**Acknowledgements. ** We would like to thank Dehua Cheng and Youwei Zhuo from USC for helpful discussions. Jian Li is supported in part by the National Basic Research Program of China Grant 2015CB358700, the National Natural Science Foundation of China Grant 61822203, 61772297, 61632016, 61761146003, and a grant from Microsoft Research Asia. Jie Tang is the corresponding author.

APPENDIX

We first prove Thm. 3.3 and Thm. 3.3 in Section 3.3. The following lemmas will be useful in our proof.

Lemma 0.

*((Trefethen and Bau III, 1997))

Singular values of a real symmetric matrix are the absolute values of its eigenvalues.*

Lemma 0.

*(Courant-Fisher Theorem)

Let $\bm{A}\in\mathbb{R}^{n\times n}$ be a symmetric matrix with eigenvalues $\lambda_{1}\geq\lambda_{2}\geq\cdots\geq\lambda_{n}$, then for $i\in[n]$,*

[TABLE]

Lemma 0.

*((Horn and Johnson, 1991))

Let $\bm{B},\bm{C}$ be two $n\times n$ symmetric matrices. Then for the decreasingly-ordered singular values $\sigma$ of $\bm{B},\bm{C}$ and $\bm{BC}$,*

[TABLE]

holds for any $1\leq i,j\leq n$ and $i+j\leq n+1$.

Lemma 0.

Let $\bm{\mathcal{L}}=\bm{D}^{-1/2}\bm{L}\bm{D}^{-1/2}$ and similarly $\widetilde{\bm{\mathcal{L}}}=\bm{D}^{-1/2}\widetilde{\bm{L}}\bm{D}^{-1/2}$. Then all the singular values of $\widetilde{\bm{\mathcal{L}}}-\bm{\mathcal{L}}$ are smaller than $2\epsilon$, i.e., $\forall i\in[n]$, $\sigma_{i}(\widetilde{\bm{\mathcal{L}}}-\bm{\mathcal{L}})<4\epsilon$.

Proof.

Notice that

[TABLE]

which is a normalized graph Laplacian whose eigenvalues lie in the interval $[0,2)$, i.e., for $i\in[n]$, $\lambda_{i}(\bm{\mathcal{L}})\in[0,2)$ (Von Luxburg, 2007). Since $\widetilde{\bm{L}}$ is a $(1+\epsilon)$-spectral sparsifier of $\bm{L}$, we know that for $\forall\bm{x}\in\mathbb{R}^{n}$,

[TABLE]

Let $\bm{x}=\bm{D}^{-1/2}\bm{y}$ which is bijective, we have

[TABLE]

The last inequality is because we assume $\epsilon<0.5$. Then, by Courant-Fisher Theorem (Lemma 2), we can immediately get, $\forall i\in[n]$,

[TABLE]

Then, by Lemma 1, $\sigma_{i}(\widetilde{\bm{\mathcal{L}}}-\bm{\mathcal{L}})<4\epsilon,\forall i\in[n]$. ∎

Given the above lemmas, we can see how the constructed $\widetilde{\bm{M}}$ approximates $\bm{M}$ and how the constructed NetMF matrix sparsifier (Eq. (6)) approximates the NetMF matrix (Eq. (3)).

\Merror

Proof.

First notice that $\widetilde{\bm{M}}-\bm{M}=\bm{D}^{-1}\left(\widetilde{\bm{L}}-\bm{L}\right)\bm{D}^{-1}=\bm{D}^{-1/2}(\widetilde{\bm{\mathcal{L}}}-\bm{\mathcal{L}})\bm{D}^{-1/2}$. Apply Lemma 3 twice and use the result from Lemma 4, we have

[TABLE]

∎

\logerror

Proof.

It is easy to observe that $\operatorname{trunc\_log^{\circ}}$ is 1-Lipchitz w.r.t. Frobenius norm. So we have

[TABLE]

∎

We finally explain the remaining question in Step 1 of NetSMF: After sampling a length-$r$ path $\bm{p}=(u_{0},\cdots,u_{r})$, why does the algorithm add a new edge to the sparsifier with weight $\frac{rm}{MZ(\bm{p})}$? Our proof relies on two lemmas from (Cheng et al., 2015b).

Lemma 0.

*(Lemma 3.3 in (Cheng et al., 2015b))

Given the path length $r$, the probability for the PathSampling algorithm to sample a path $\bm{p}$ is $\tau(\bm{p})=\frac{w(\bm{p})Z(\bm{p})}{2rm}$, where $Z(\bm{p})$ is defined in Eq. (7) and*

[TABLE]

Lemma 0.

*(Theorem 2.2 in (Cheng et al., 2015b))

After sampling a length-$r$ path $\bm{p}=(u_{0},u_{1},\cdots,u_{r})$, the weight corresponding to the new edge $(u_{0},u_{r})$ added to the sparsifier should be $\frac{w(\bm{p})}{\tau(\bm{p})M}$.*

Theorem 7.

After sampling a length-$r$ path $\bm{p}=(u_{0},u_{1},\cdots,u_{r})$ using the PathSampling algorithm (Alg. 2). The weight of the new edge added to the sparsifier $\widetilde{\bm{L}}$ is $\frac{2rm}{MZ(\bm{p})}$.

Proof.

The proof is to plug the definition of $Z(\bm{p})$, $w(\bm{p})$, and $\tau(\bm{p})$ from Lemma 5 into Lemma 6, that is,

[TABLE]

For unweighted networks, this weight can be simplified to $\frac{m}{M}$, since $Z(\bm{p})=2r$ for unweighted networks. ∎