Statistical guarantees for local graph clustering

TL;DR

This paper provides statistical guarantees for the l1-regularized PageRank method in local graph clustering, demonstrating its ability to accurately recover target clusters with bounded false positives and exact recovery under certain conditions, supported by empirical and theoretical analysis.

Contribution

It offers the first statistical analysis of l1-regularized PageRank for cluster recovery, proving support recovery guarantees, computational efficiency, and equivalence with APPR.

Findings

Supports full cluster recovery with bounded false positives

Achieves exact cluster recovery when seed is connected only to target

Demonstrates state-of-the-art empirical performance

Abstract

Local graph clustering methods aim to find small clusters in very large graphs. These methods take as input a graph and a seed node, and they return as output a good cluster in a running time that depends on the size of the output cluster but that is independent of the size of the input graph. In this paper, we adopt a statistical perspective on local graph clustering, and we analyze the performance of the l1-regularized PageRank method~(Fountoulakis et. al.) for the recovery of a single target cluster, given a seed node inside the cluster. Assuming the target cluster has been generated by a random model, we present two results. In the first, we show that the optimal support of l1-regularized PageRank recovers the full target cluster, with bounded false positives. In the second, we show that if the seed node is connected solely to the target cluster then the optimal support of l1-regularized PageRank recovers exactly the target cluster. We also show empirically that l1-regularized PageRank has a state-of-the-art performance on many real graphs, demonstrating the superiority of the method. From a computational perspective, we show that the solution path of l1-regularized PageRank is monotonic. This allows for the application of the forward stagewise algorithm, which approximates the solution path in running time that does not depend on the size of the whole graph. Finally, we show that l1-regularized PageRank and approximate personalized PageRank (APPR), another very popular method for local graph clustering, are equivalent in the sense that we can lower and upper bound the output of one with the output of the other. Based on this relation, we establish for APPR similar results to those we establish for l1-regularized PageRank.

Equations

$$p^{\text{PR}}=\alpha \mathbf{s}+(1-\alpha )W{p}^{\text{PR}},$$

$$\hat{x}=\underset{x\in {\mathbb{R}}^n}{\mathrm{argmin}}\left\{\underset{:=f(x)}{\underbrace{\frac{1}{2}{x}^{\top }Qx-\alpha x^{\top }\mathbf{s}}}+\rho \alpha \Vert Dx{\Vert }_1\right\},$$

$${\nabla }_{i}f(\hat{x})=(Q\hat{x}{)}_i-\alpha \mathbf{s}{}_i=$$

$$\{\begin{array}{l}\text{Choose an i∈[n] such that ∇if(x(k))≤−ραdi};\\ \text{Update xi(k+1)=xi(k)−di−1∇if(x(k)). }\end{array}$$

$$\mathrm{support}(\hat{x}({\rho }_0))\subseteq \mathrm{support}(x^{\text{APPR}}({\rho }_0))\subseteq \mathrm{support}(\hat{x}((1-\alpha )\cdot {\rho }_0/2)).$$

$$\mathbb{E}\left[A_{ij}\right]=\{\begin{array}{ll}p& \text{if i∈K and j∈K},\\ q& \text{if i∈K and j∈Kc},\\ \text{Any value}& \text{if i∈Kc and j∈Kc}.\end{array}$$

$$\overline{\text{Cond}}=1-\gamma ,\text{ where }\gamma :=\frac{p\cdot (k-1)}{\bar{d}}\in (0,1).$$

$$\rho (\delta ):={\left(\frac{1-\alpha }{1+\alpha }\right)}^2{\left(\frac{1-\delta }{1+\delta }\right)}^2\frac{\gamma p}{(1+\delta ){\bar{d}}^2}=\mathcal{O}\left(\frac{\gamma p}{{\bar{d}}^2}\right),$$

$$K\subseteq \mathrm{support}(\hat{x}),$$

$$\textnormal{Vol}(\text{FP})\leq\textnormal{Vol}(K)\big{[}\underbrace{\left(\frac{1+\alpha}{1-\alpha}\right)^{2}\left(\frac{1+\delta}{1-\delta}\right)^{3}\frac{1}{\gamma^{2}}-1}_{=\mathcal{O}\left({\frac{1}{\gamma^{2}}}\right)-1}\big{]},$$

$$\mathrm{support}(\hat{x})\subseteq K,$$

$$\frac{C(0.5c+1)}{\gamma p}=\mathcal{O}\left(\frac{1}{\gamma p}\right)<d_j,$$

$$\mathrm{support}(\hat{x})=K,$$

$$K\subseteq \mathrm{support}(x^{\text{APPR}})\text{ and }\text{Vol}(\text{FP}(x^{\text{APPR}}))\le \text{Vol}(K)\left[\mathcal{O}\left(\frac{1}{{\gamma }^2}\right)-1\right],$$

$$\mathrm{support}(x^{\text{APPR}})=K,$$

$$\mathcal{O}\left(\frac{1}{\gamma p}\right)<d_j,$$

$$\{\begin{array}{l}\text{Choose an i∈[n] such that di−1∇if(x(t))≤0 is the smallest among [n]};\\ \text{Update xi(k+1)=xi(k)+di−1η. }\end{array}$$

$$\sigma =\min \left[\max \left(\frac{\text{Vol}(R)}{\text{Vol}(V-R)},\frac{\text{Vol}(R\cap K)}{\text{Vol}(K)}\right),1\right].$$

$$\text{For all i∈K: }\left|X_i-\mu \right|\le \delta \mu .$$

$$\text{For all i∈K: }\left|d_i-\bar{d}\right|\le \delta \bar{d},$$

$$\underset{j\in K^c}{\max }\Vert A_{j,K}{\Vert }_1\le c+2.$$

$$x^{\star }=\underset{x}{\mathrm{argmin}}\left\{\frac{1}{2}{x}^{\top }\mathbb{E}\left[Q\right]x-\alpha x^{\top }\mathbf{s}+\rho \alpha \Vert \mathbb{E}\left[D\right]x{\Vert }_1\right\}.$$

$$\mathrm{support}(x^{\star })=K,$$

$${\rho }^{♯}=\frac{q(1-\alpha )}{2\alpha \bar{d}\cdot \mathbb{E}\left[d_{{\ell }_{♯}}\right]+q(1-\alpha )\left[k\bar{d}+(n-k)\mathbb{E}\left[d_{{\ell }_{♯}}\right]\right]},\text{ and }{\rho }^{♮}=\frac{p(1-\alpha )}{\bar{d}\left[(1+\alpha )\bar{d}+(1-\alpha )p\right]},$$

$$\{\begin{array}{l}u=\frac{2\alpha }{(1+\alpha )\bar{d}+(1-\alpha )p};\\ v=\frac{\frac{1-\alpha }{2}p\cdot u-\rho \alpha \bar{d}}{\alpha \bar{d}+\frac{1-\alpha }{2}q(n-k)}.\end{array}$$

$${\hat{x}}^{\text{R}}$$

$$=\underset{y\in {\mathbb{R}}^k}{\mathrm{argmin}}\left\{\frac{1}{2}{y}^{\top }{Q}_{K,K}y-\alpha y^{\top }{\mathbf{s}}_K+\rho \alpha \Vert D_{K,K}y{\Vert }_1\right\}.$$

$$\mathrm{support}({\hat{x}}^{\text{R}})=K,$$

$$\mathrm{support}({\hat{x}}^{\text{R}})\subseteq \mathrm{support}(\hat{x}).$$

$$\Vert {\hat{x}}_{K\backslash S}^{\text{R}}{\Vert }_{\infty }\le \underset{=x_{K\backslash S}^{\star }}{\underbrace{v}}+\underset{\ge \Vert {\hat{x}}_{K\backslash S}^{\text{R}}-x_{K\backslash S}^{\star }{\Vert }_{\infty }}{\underbrace{\frac{c_1(1-\alpha )u}{\bar{d}}+2c_1\delta (v+\rho \alpha )}},$$

$$\{\begin{array}{ll}{\hat{x}}_i({\rho }_0)<x_i^{\text{APPR}}({\rho }_0),& \text{i∈support(xiAPPR(ρ0)),}\\ {\hat{x}}_i({\rho }_0)=x_i^{\text{APPR}}({\rho }_0)=0,& \text{i∈/support(xiAPPR(ρ0)),}\end{array}$$

Full text

Keywords: clustering, local graph clustering, Page-Rank, seed expansion, $\ell_{1}$-regularization.

1 Introduction

In many data applications, one is interested in finding small-scale structure in a very large data set. As an example, consider the following version of the so-called local graph clustering problem: given a large graph and a seed node in that graph, quickly find a good small cluster that includes that seed node. From an algorithmic perspective, one typically considers worst-case input graphs, and one is interested in running time guarantees, e.g., to find a good cluster in a time that depends linearly or sub-linearly on the size of the entire graph. From a statistical perspective, such a local graph clustering problem can be understood as a recovery problem. One assumes that there exists a target cluster in a given large graph, where the graph is assumed to have been generated by a random model, and the objective is to recover the target cluster from one node inside the cluster.

In this paper, we consider the so-called $\ell_{1}$-regularized PageRank algorithm (Fountoulakis et al., 2019), a popular algorithm for the local graph clustering problem, and we establish statistical recoverability guarantees for it. Previous theoretical analysis on local graph clustering, e.g., Andersen et al. (2006); Zhu et al. (2013), is based on the notion of conductance (a cluster quality metric that considers the internal versus external connectivity of a cluster) and considers running time performance for worst-case input graphs. In contrast, our goal will be to study the average-case performance of the $\ell_{1}$-regularized PageRank algorithm, under a certain type of a local random graph model. The model we consider is very general; it concerns the target cluster and its adjacent nodes; and it encompasses the stochastic block model (Holland et al., 1983; Abbe, 2017) and the planted clustering model (Alon et al., 1998; Arias-Castro and Verzelen, 2014) as special cases.

Within this random graph model, we provide theoretical guarantees for the unique optimal solution of the $\ell_{1}$-regularized PageRank optimization problem. In particular, the cluster is recovered through the support set of the $\ell_{1}$-regularized PageRank vector and we give rigorous bounds on the false positives and false negatives of the recovered cluster. Furthermore, observe that our statistical perspective is more aligned with statistical guarantees for the sparse regression problem (and the lasso problem (Tibshirani, 1996)), where the objective is to recover the true parameter and/or support from noisy data. Given this connection, we also establish a result for the exact support recovery of $\ell_{1}$-regularized PageRank. Empirically we demonstrate the ability of the method to recover the target cluster in a range of real-world data graphs. Finally, we establish an equivalence between $\ell_{1}$-regularized PageRank and the very popular local graph clustering algorithm from Andersen et al. (2006); Zhu et al. (2013). This allows us to prove similar average case guarantees for the algorithm of Andersen et al. (2006); Zhu et al. (2013) as well.

1.1 Literature review

The origins of local graph clustering are with the work of Spielman and Teng (2013). Subsequent to their original results, there has been a great deal of follow-up work on local graph clustering procedures, including with random walks (Andersen et al., 2006), local Lanczos spectral approximations (Shi et al., 2017), evolving sets (Andersen et al., 2016), seed expansion methods (Kloumann and Kleinberg, 2014), optimization-based approaches (Fountoulakis et al., 2019, 2017), and local flow methods (Wang et al., 2017). There also exist local higher-order clustering (Yin et al., 2017), linear algebra approaches (Shi et al., 2017), spectral methods based on Heat Kernel PageRank (Kloster and Gleich, 2014), newer seed set selection techniques for local flow methods (Veldt et al., 2019), and parallel local spectral approaches (Shun et al., 2016). In all of these cases, given a seed node, or a seed set of nodes, the goal of existing local graph clustering approaches is to compute a cluster “nearby” the seed that is related to the “best” cluster nearby the seed. Here, “best” and “nearby” are intentionally left under-specified, as they can be formalized in one of a few different but related ways. For example, “best” is usually related to a clustering score such as conductance. In fact, many existing methods for local graph clustering with theoretical guarantees are motivated through the problem of finding a cluster that is near the seed node and that also has small conductance value (Spielman and Teng, 2013; Andersen et al., 2006; Fountoulakis et al., 2019; Veldt et al., 2019; Wang et al., 2017). Recently, Green et al. (2019) studied local graph clustering from a traditional statistical learning setup to identify density clusters.

There are also numerous papers in statistics on partitioning random graphs. Arguably, the stochastic block model (SBM) is the most commonly employed random model for graph partitioning, and it has been extensively studied (Abbe and Sandon, 2015; Abbe et al., 2015; Zhang and Zhou, 2016; Massoulié, 2014; Mossel et al., 2018; Newman et al., 2002; Mossel et al., 2015; Rohe et al., 2011; Amini and Levina, 2018; Abbe, 2017). Recent work has also generalized the SBM to a degree-corrected block model, to capture degree heterogeneity of the network (Chen et al., 2018; Gulikers et al., 2017; Zhao et al., 2012; Gao et al., 2018). The literature in this area is too extensive to cover in this paper, but we refer the readers to excellent survey papers on the graph partitioning problem (Abbe, 2017).

We should emphasize that the traditional graph partitioning problem is quite different than the local graph clustering problem that we consider in this paper. Among other things, the former partitions all the vertices of a graph into different clusters, while for the latter problem our objective is to find a single cluster given a seed node in the cluster; the former takes as input a graph, while the latter takes as input a graph and a seed set of nodes; and the former runs in time depending on the size of the graph, while the latter runs in time depending on the size of the output, but is otherwise independent of the size of the graph.

1.2 Notation

We write $[n]=\{1,\dots,n\}$ for any $n\geq 1$. Throughout the paper we assume we have a connected, undirected graph $G=(V,E)$, where $V$ denotes the set of nodes, with $\left\lvert{V}\right\rvert=n$, and $E\subset(V\times V)$ denotes the set of edges. We denote by $A$ the adjacency matrix of $G$, i.e., $A_{ij}=w_{ij}$ if $(i,j)\in E$, and [math] otherwise. For an unweighted graph, $w_{ij}$ is set to $1$ for all $(i,j)\in E$. We denote by $D$ the diagonal degree matrix of $G$, i.e., $D_{ii}\coloneqq d_{i}=\sum_{j:(i,j)\in E}w_{ij}$, where $d_{i}$ is the weighted degree of node $i$. In this case, $d=(d_{i})\in\mathbb{R}^{n}$ denotes the degree vector, and the volume of a subset of nodes is defined as $\textnormal{Vol}(B)=\sum_{i\in B}d_{i}$ for $B\subseteq V$. We denote by $L=D-A$ the graph Laplacian; and $Q\coloneqq\alpha D+\frac{1-\alpha}{2}L$.

For given sets of indexes $B_{1},B_{2}\subseteq[n]$, we write $M_{B_{1},B_{2}}$ to denote the submatrix of $M$ indexed by $B_{1}$ and $B_{2}$. If $B_{1}=\{i\}$ is a singleton, we use $M_{i,B_{2}}$ to indicate the $i$-th row of $M$ whose columns are indexed by $B_{2}$. Analogously, we use $M_{B_{1},j}$ to indicate the $j$-th column of $M$ whose rows are indexed by $B_{1}$. We denote by $B_{1}\backslash B_{2}$ a set difference between $B_{1}$ and $B_{2}$, and denote by $B_{1}^{c}=[n]\backslash B_{1}$ the complement of $B_{1}$.

2 Local graph clustering from a variational point of view

In this section, we briefly review and motivate $\ell_{1}$-regularized PageRank (Fountoulakis et al., 2019), an optimization formulation of local graph clustering to find a local cluster around a given seed node. We study some properties of the output vector of $\ell_{1}$-regularized PageRank and illuminate a novel connection to approximate personalized PageRank (APPR) (Andersen et al., 2006).

2.1 Background on $\ell_{1}$-regularized PageRank

PageRank (Page et al., 1999; Brin and Page, 1998) is a popular approach for ranking the importance of the nodes given a graph. It is defined as the stationary distribution of a Markov chain, which is encoded by a convex combination of the input distribution $\mathbf{s}\in\mathbb{R}^{n}$ and the (lazy) random walk on the graph, i.e.,

[TABLE]

where $W=(I+AD^{-1})/2$ is the lazy random walk operator and where $\alpha\in(0,1)$ is the teleportation parameter. To measure the ranking or importance of the nodes of the “whole” graph, PageRank is often computed by setting the input vector $\mathbf{s}$ to be a uniform distribution over $\{1,2,\ldots,n\}$.

For local graph clustering, where the aim is to identify a target cluster, given a seed node in the cluster, the input distribution $\mathbf{s}$ is set to be equal to one for the seed node and zero everywhere else. For example, when the node $i$ is given as the seed node, we consider the input distribution $\mathbf{s}$ to be the discrete Dirac measure such that $\mathbf{s}_{i}=1$ and zero elsewhere. This “personalized” PageRank (Haveliwala, 2002) measures the closeness or similarity of the nodes to the given seed node, and it outputs a ranking of the nodes that is “personalized” with respect to the seed node (as opposed to the original PageRank, which considers the entire graph). From an operational point of view, the underlying diffusion process in (1) defining personalized PageRank performs a lazy random walk with probability $1-\alpha$ and “teleports” a random walker back to the original seed node with probability $\alpha$.

From the definition itself, the personalized PageRank vector can be obtained by solving the linear system (1). Unfortunately, this step can be prohibitively expensive, especially when there is a single seed node or a small seed set of seed nodes, and when one is interested in very small clusters in a very large graph. In the seminal work of Andersen et al. (2006), the authors propose an iterative algorithm, called approximate personalized PageRank (APPR), to solve this running time problem. They do so by approximating the personalized PageRank vector, while running in time independent of the size of the entire graph. APPR was developed from an algorithmic (or “theoretical computer science”) perspective, but it is equivalent to applying a coordinate descent type algorithm to the linear system (1) with a particular scheme of early stopping (see Section 2.3 for more details on the APPR algorithm). Motivated by this, Fountoulakis et al. (2019) proposed the $\ell_{1}$-regularized PageRank optimization problem. Unlike APPR, the solution method for the $\ell_{1}$-regularized PageRank optimization problem is purely optimization-based. It uses an $\ell_{1}$ norm regularization to set automatically to be zero all nodes that are dissimilar to the seed node, thereby resulting in a highly sparse output. In this manner, $\ell_{1}$-regularized PageRank can estimate the personalized ranking, while maintaining the most relevant nodes at the same time. Prior work (Fountoulakis et al., 2019) also showed that proximal gradient descent (ISTA) can solve the $\ell_{1}$-regularized PageRank minimization problem, with access to only a small portion of the entire graph, i.e., without even touching the entire graph, therefore allowing the method to easily scale to very large-scale graphs (Shun et al., 2016).

In this paper, we investigate the statistical performance of $\ell_{1}$-regularized PageRank by reformulating the local graph clustering into the problem of sparse support recovery. Here we give a more precise definition of the $\ell_{1}$-regularized PageRank optimization problem from Fountoulakis et al. (2019) that we consider.

Definition 1** ($\ell_{1}$-regularized PageRank)**

Given a graph $G=(V,E)$, with $|V|=n$, and a seed vector $\mathbf{s}\in\mathbb{R}^{n}$, the $\ell_{1}$-regularized PageRank vector on the graph is defined as

[TABLE]

where recall $Q=\alpha D+\frac{1-\alpha}{2}L$, and where $\rho>0$ is a user-specified parameter that controls the amount of the regularization.

Note that our definition of $\ell_{1}$-regularized PageRank is consistent with the original formulation (Fountoulakis et al., 2019, Equation (8)) by change of variables $D^{1/2}x=q$. To better understand the objective function of (2), when the regularization parameter $\rho$ is set to [math], one can easily check that the re-scaled version of the output solution $D\widehat{x}$ recovers the original PageRank solution, that is, $D\widehat{x}=p^{\text{PR}}$ satisfies the stationary equation (1). For the stationary personalized PageRank vector $p^{\text{PR}}$, mass is concentrated around the seed node, meaning that after ordering, it has long tail for nodes far away from the seed node. Importantly, we can then efficiently cut this tale using $\ell_{1}$ norm regularization, without even having to compute the long tail.

2.2 Properties

Here, we state some properties of the $\ell_{1}$-regularized PageRank vector that will be useful for our analysis, as well as for gaining further insight into the method. The proof of these lemmas can be found in Appendix C.

First, the following lemma guarantees that the $\ell_{1}$-regularized PageRank vector is non-negative. This result should be natural, because $\ell_{1}$-regularized PageRank computes the importance scores of the nodes relative to the seed node, which cannot be negative.

Lemma 2** (Non-negativity of $\ell_{1}$-reg. PageRank.)**

Let $\widehat{x}$ be the optimal output solution given in Definition 1. Then $\widehat{x}$ is non-negative, i.e., $\widehat{x}_{j}\geq 0$ for all $j\in V$.

The next lemma guarantees that the gradient of $f$ at the optimal solution $\widehat{x}$ must be non-positive. Since the $\ell_{1}$-regularized PageRank problem is strongly convex (the minimum eigenvalue of $Q$ is $>0$), by KKT condition, this characterization is both necessary and sufficient for $\widehat{x}$ to be the unique solution. We frequently use this lemma in the proof of our subsequent results.

Lemma 3** (Optimality condition for $\ell_{1}$-reg. PR minimization.)**

Let $\operatorname{support}(\widehat{x}):=\{i\in V\ |\ \widehat{x}_{i}\neq 0\}$ be the support set of the optimal solution. Then

[TABLE]

Finally, the next result shows that the solution path of the $\ell_{1}$-regularized PageRank problem is monotone, meaning that when the output of $\ell_{1}$-regularized PageRank is parametrized as a function of $\rho>0$, the trajectory $\widehat{x}(\rho)$ changes in a monotonic manner as $\rho$ varies.

Lemma 4** (Monotonicity of $\ell_{1}$-reg. PageRank.)**

Let $\widehat{x}(\rho)$ denote the solution for (2) indexed by $\rho>0$. Then, $\widehat{x}(\rho)$ is monotone as a function of $\rho$, i.e., $\widehat{x}(\rho_{0})\leq\widehat{x}(\rho_{1})$ whenever $\rho_{0}>\rho_{1}$, where $\leq$ is applied component-wise.

Lemma 4 shows that once a node enter the model at some number $\rho>0$, then it will never leave the model thereafter. This result is intuitive since the importance score of the nodes will increase as the amount of regularization or shrinkage decreases. In Section 4, we will use this monotonic property in a crucial way to develop an iterative algorithm that approximates the entire solution path.

2.3 Connection to approximate personalized PageRank (APPR)

Approximate personalized PageRank (APPR), first studied in Andersen et al. (2006) and followed up by a number of subsequent works, is at the heart of local graph clustering whose central idea is to approximate the original PageRank vector without ever touching the entire graph. Fountoulakis et al. (2019) observed that the APPR algorithm is equivalent to the iterative coordinate descent solver applied to the linear system (1) with a specifically designed termination criterion. In our notation, we can write the algorithm in a simple and compact way:

• •

Given a parameter $\rho>0$, initialize $x^{(0)}=0$;

• •

For each $k\geq 0$, while $\lVert{D^{-1}\nabla f(x^{(k)})}\rVert_{\infty}\geq\rho\alpha$, iterate the steps111It can be shown that if APPR is initialized at $x^{(0)}=0$ then $\nabla f(x^{(k)})\leq 0$ for all $k\geq 0$. In this case, the termination criterion of APPR reads as $\nabla_{i}f(x^{(k)})>-\rho\alpha d_{i}$ for all $i\in[n]$.

[TABLE]

• •

Output $x^{(k^{\star})}=:x^{\text{APPR}}$.

The output of the APPR algorithm is the personalized score vector $x^{\text{APPR}}$ that efficiently approximates the original PageRank. Note that the updating formula (3) requires one evaluation of the gradient which can be done by accessing only neighboring nodes of the selected entry $i$. As a result the APPR algorithm can solve the PageRank equation (1) extremely efficient; and in particular the running time of the algorithm depends on the size of the output rather than the size of the whole graph.

APPR was developed purely from an algorithmic perspective and the output of the algorithm depends on which coordinate $i$ is chosen at every iteration (see the updating step (3)). This property, while allowing the algorithm to enjoy the locality property and resulting in the output vector that is highly sparse, can lead to different results of local clustering in principle (albeit the results may look more or less similar). On the other hand, $\ell_{1}$-regularized PageRank eliminates this issue by decoupling the locality/sparsity of the local clustering output vector from the algorithmic issue of running in time independent of the size of the graph. In particular, any convex optimization algorithm that exploits the locality of the problem, such as proximal gradient descent (ISTA), can be employed to minimize the $\ell_{1}$-regularized objective function (2) while touching only the neighbors of the current non-zero nodes. Therefore the formulation via $\ell_{1}$-regularized PageRank achieves two objectives of the graph processing that are desirable for local graph clustering, i.e., both the locality/sparsity of the solution and the “strongly local” running time of the algorithm.

It is shown in Fountoulakis et al. (2019) that $\ell_{1}$-regularized PageRank can be viewed as a variational formulation of APPR, in the sense that the optimality condition for the $\ell_{1}$-regularized objective function (Lemma 3) implies the termination criterion of APPR, i.e., $\lVert{D^{-1}\nabla f(x^{(k)})}\rVert_{\infty}<\rho\alpha$. In this work, we strengthen the connection between the output of APPR and the output of $\ell_{1}$-regularized PageRank. In particular, we show that with appropriate choices of the regularization parameters, both approaches become equivalent in terms of cluster recovery.

Theorem 5** (Equivalence between $\ell_{1}$-reg. PR and APPR.)**

Let $\widehat{x}(\rho)$ be the $\ell_{1}$-regularized PageRank vector (2), let $x^{\text{APPR}}(\rho)$ be the output of APPR (3) at the regularization parameter $\rho$. Then for any $\rho_{0}>0$,

[TABLE]

That is, for any parameter $\rho=\rho_{0}>0$, the support set of the output of APPR is, respectively, a superset and a subset of that of $\ell_{1}$-regularized PageRank at $\rho=\rho_{0}$ and $\rho=(1-\alpha)\cdot\rho_{0}/2$.

Theorem 5 establishes rather a stronger equivalence between the $\ell_{1}$-regularized PageRank and the APPR approaches than was established in the prior work of Fountoulakis et al. (2019). Importantly, using this result, various statistical guarantees for the output cluster of APPR directly follow from the results we establish for the output of $\ell_{1}$-regularized PageRank. While analyzing APPR’s output in the random graph setting involves technical challenges due to its algorithmic nature, the fact that the output cluster of $\ell_{1}$-regularized PageRank is given by the optimization solution allows us to analyze statistical properties more easily. In the next section, we will study the recovery guarantees for $\ell_{1}$-regularized PageRank under a random graph model, and we will show how this guarantee can be transferred to the output cluster of APPR using Theorem 5.

3 Statistical guarantees under random model

We begin by introducing a random model that we consider for generating a target cluster. In particular we will assume the graph is generated according to the following model.

Definition 6** (Local random model.)**

Given a graph $G=(V,E)$ that has $n$ vertices, let $K\subseteq V$ be a target cluster inside the graph, and let $K^{c}$ denote the complement of $K$. If two vertices $i$ and $j$ belong to $K$, then we draw an edge between $i$ and $j$ with probability $p$, independently of all other edges; if $i\in K$ and $j\in K^{c}$, then we draw an edge with probability $q$, independently of all other edges; and otherwise, we allow any (deterministic or random) model to generate edges among vertices in $K^{c}$.

According to Definition 6, the adjacency matrix $A\in\mathbb{R}^{n\times n}$ is symmetric, and for any $i,j\in V$, we have that $A_{ij}$ is an independent draw from a Bernoulli distribution with probability $p$ if $i,j\in K$, and from a Bernoulli distribution with probability $q$ if $i\in K$ and $j\in K^{c}$. For the rest of the graph, i.e., when both $i$ and $j$ belong to $K^{c}$, $A_{ij}$ can be generated from an arbitrary fixed model. Under this definition, we can also naturally define the population version of the graph, which is the graph induced by the expected adjacency matrix $\mathbb{E}\left[{A}\right]$, where the expectation is taken with respect to the distribution defined by our random model. That is, the population graph is an undirected graph $\overline{G}=(V,E)$ whose adjacency matrix is $\mathbb{E}\left[{A}\right]$, where

[TABLE]

The expected degree matrix is similarly denoted by $\mathbb{E}\left[{D}\right]$ and the expected graph Laplacian is defined as $\mathbb{E}\left[{L}\right]=\mathbb{E}\left[{D}\right]-\mathbb{E}\left[{A}\right]$. The model in Definition 6 allows us to formulate the problem of local graph clustering as the recovery of a target cluster. Since we are interested in recovering a single target cluster, it is natural to make assumptions only for nodes in the target cluster and nodes adjacent to the target cluster, and to leave the interactions between other nodes unspecified.

This random model is fairly general, and it covers several popular random graph models appearing in the literature, including the stochastic block model (SBM) (Holland et al., 1983; Abbe, 2017) and the planted clustering model (Alon et al., 1998; Arias-Castro and Verzelen, 2014; Chen and Xu, 2016). For instance, if the subgraph with the vertices within $K^{c}$ is generated from the SBM, then the entire graph $G=(V,E)$ follows the SBM. On the other hand, if the subgraph of $K^{c}$ is generated from the classical Erdős-Rényi model with probability $q$, the entire graph $G=(V,E)$ follows the Planted Densest Subgraph (in this case nodes in $K^{c}$ do not belong to any clusters). Hence, the results we obtain here for our model holds more broadly across these different random graph models.

Before we move on to our results, we need additional piece of notation. We write $S\subseteq K$ to denote a singleton of the given seed node. Let $k=\left\lvert{K}\right\rvert$ denote the cardinality of the target cluster. According to our local model, any node in the target cluster has the same expected degree, $\mathbb{E}\left[{d_{i}}\right]=p(k-1)+q(n-k)$ for all $i\in K$, which we denote by $\bar{d}$. For the nodes $\ell$ outside $K$, we write $\mathbb{E}\left[{d_{\ell}}\right]$ to denote its expected degree, where the expectation is taken with respect to a distribution that generates the graph in $K^{c}$. Conductance measures the weight of the edges that are being removed over the volume of the cluster—formally it is defined as the ratio $\mbox{Cut}(S,S^{c})/\min\left(\textnormal{Vol}(S),\textnormal{Vol}(S^{c})\right)$, where $\mbox{Cut}(S,S^{c}):=\sum_{i\in S,j\in S^{c}}A_{ij}$. From Definition 6, the conductance of the target cluster of the population graph $\overline{G}$ is given by

[TABLE]

Here $\gamma$ can be viewed as the ratio of the random walker staying inside $K$ under the population graph. (Note $\bar{d}$ is the expected degree of the target cluster and $p\cdot(k-1)$ is the expected degree of the target cluster when restricted to the subgraph within $K$.)

As in the worst-case analysis of local graph clustering (Andersen et al., 2006; Zhu et al., 2013), conductance $\overline{\text{Cond}}$, or equivalently the number $\gamma$, will play a crucial role in determining the behavior of $\ell_{1}$-regularized PageRank under the local graph model. In particular, we see that in the extreme scenario where $\gamma=1$, we have $q=0$ indicating perfect separability of the target cluster from the rest, while for $\gamma=0$, we have $p=0$ meaning there is no signal to ever recover. With this definition, we can also write $p(k-1)=\gamma\bar{d}$ and $q(n-k)=(1-\gamma)\bar{d}$.

3.1 Recovery of target cluster with bounded false positives

Here, we investigate the performance of $\ell_{1}$-regularized PageRank on the graph generated by the local random model as in Definition 6, and we state two of our main theorems.

Our first main result guarantees full recovery of the target cluster for an appropriate choice of the regularization parameter. Specifically, for $\delta>0$, define

[TABLE]

where $\alpha$ is the teleportation constant and where $p,\gamma,\bar{d}$ are the parameters of the random model defined in (5). Then, if we solve the convex problem (2) with $\rho\leq\rho(\delta)$, the optimal solution fully recovers the target cluster $K$, as long as the seed node is initialized inside $K$.

Theorem 7** (Full recovery.)**

Suppose that $p^{2}k\geq\mathcal{O}\left({\delta^{-2}\log k}\right)$. If the regularization parameter satisfies $\rho\leq\rho(\delta)$ where $\rho(\delta)$ is defined in (6), then the solution to Problem (2) fully recovers the cluster $K$, i.e.,

[TABLE]

with probability at least $1-6\exp(-\mathcal{O}\left({\delta^{2}p^{2}k}\right))$.222More precisely, we assume $(1-\delta)p^{2}k\geq c_{0}^{-1}\delta^{-2}\log k$ for a fixed constant $c_{0}>0$. Then with probability at least $1-6e^{-c_{0}\delta^{2}(1-\delta)p^{2}k}$, the statement in the theorem holds.

Our next main result provides an upper bound on the false positives present in the support set of the $\ell_{1}$-regularized PageRank vector. By “false positives”, we mean the nonzero nodes that belong to $K^{c}$. We measure the size of false positives using a notion of volume, where we recall the volume of a subset of vertices $B\subseteq V$ is given by $\textnormal{Vol}(B)=\sum_{i\in B}d_{i}$.

Theorem 8** (Bounds on false positives.)**

Suppose the same conditions as in Theorem 7. If the regularization parameter satisfies $\rho\geq\rho(\delta)$, then the solution to Problem (2) satisfies the bound

[TABLE]

with probability at least $1-6\exp(-\mathcal{O}\left({\delta^{2}p^{2}k}\right))$,333The same probability bound holds as in Theorem 7. where $\text{FP}=\{i\in\operatorname{support}(\widehat{x}):i\in K^{c}\}$ is the collection of false positive nodes.

The above results, Theorem 7 and Theorem 8, show several regimes where $\ell_{1}$-regularized PageRank can fully recover the target cluster with nonvanishing probability. In particular, when $p=\mathcal{O}\left({1}\right)$, the size of the target cluster, $k$, is required to be larger than $\mathcal{O}\left({\log k}\right)$, which includes the constant size $k=\mathcal{O}\left({1}\right)$. This is often the regime of interest for local graph clustering, where the goal is to find small- and meso-scale clusters in massive graphs (Leskovec et al., 2009, 2010). In addition, Theorem 7 indicates that if $\gamma$ is small, we need to set $\rho$ to be small to recover the entire cluster. Intuitively, more mass will leak out to $K^{c}$ for small $\gamma$, so we need to run more steps of random walk (equivalently a smaller $\rho$ in our optimization framework) to find the right cluster. However, this means that the $\ell_{1}$-regularized PageRank vector will also pick up many nonzero nodes in $K^{c}$, resulting in many false positives in the support set. Indeed, Theorem 8 shows that the volume of false positives grows quadratically as $1/\gamma$, so we need $\gamma$ to be well-bounded to get a meaningful recovery from local clustering. In the case of $p=\mathcal{O}\left({1}\right),k=\mathcal{O}\left({1}\right)$, this amounts to requiring that $q=\mathcal{O}\left({\frac{1}{n}}\right)$ in order for the recovered cluster to keep high mass inside $K$.

It is also worth making several other comments regarding the results. First, we suspect the current bound we obtain in (7) may not be tight with respect to $\alpha$ and other constants, and especially the factor $(\frac{1+\alpha}{1-\alpha})^{2}$ may be an artifact of our proof. Studying the lower bound on the performance of the method, as well as obtaining an improved bound on false positives, is therefore an interesting future direction to pursue. Furthermore, on the basis of our empirical results, $\ell_{1}$-regularized PageRank performs well across a broad range of $\alpha$ values, and we have not seen much difference in terms of performance among different $\alpha$’s. The role of $\alpha$ in $\ell_{1}$-regularized PageRank is closely tied to the regularization parameter $\rho$, and we leave the question of selecting optimal $\alpha$ for future work.

3.2 Exact recovery of target cluster with no false positives

Next, we study the scenarios under which $\ell_{1}$-regularized PageRank can exhibit a stronger recovery guarantee. Specifically, under some additional conditions, we show that the support set of the optimal solution (2) identifies the target cluster exactly, without making any false positives. For this stronger exact recovery result, we require the following assumption about the parameters of the model.

Assumption 1

We assume $p=\mathcal{O}\left({1}\right),k=\mathcal{O}\left({1}\right)$, i.e., the within-cluster connectivity and the size of the target cluster do not scale with the size of the graph $n$. Also, we assume $q=\frac{c}{n}$ for a fixed numerical constant $c>0$.

As we noted previously, the setting $k=\mathcal{O}\left({1}\right)$ is often the case of interest for local graph clustering, where we would like to identify small- and medium-scale structure in large graphs (Leskovec et al., 2009, 2010). In this case, Assumption 1 requires $p=\mathcal{O}\left({1}\right)$, so that the underlying “signal” of the problem does not vanish as the size of the graph grows, $n\to\infty$. As discussed earlier, this means $q$ must also scale as $\mathcal{O}\left({n^{-1}}\right)$ for the local clustering algorithm to find the target without making many false positives.

Now we turn to the statement of exact recovery guarantees for $\ell_{1}$-regularized PageRank when applied to the noisy graph generated from Definition 6. In particular, the fact that $q=\mathcal{O}\left({n^{-1}}\right)$ from Assumption 1 allows that with nonvanishing probability there is a node in the target cluster that is solely connected to $K$. This node will serve as a “good” seed node input in the $\ell_{1}$-regularized PageRank. With this choice of seed node, we now give conditions under which the optimal solution $\widehat{x}$ has no false positives with nonvanishing probability.

Theorem 9** (No false positives.)**

Suppose the same conditions as in Theorem 7. Assume also that Assumption 1 holds and that the size of the target cluster $k$ is $\geq 2(c+3)$. Fix $\delta\leq 0.1$. If the regularization parameter satisfies $\rho\geq\rho(\delta)$, then there is a good starting node in $K$ such that the solution to Problem (2) with that node as a seed node and with teleportation parameter $\alpha\in[0.1,0.9]$ satisfies

[TABLE]

with probability at least $1-6\exp(-\mathcal{O}\left({\delta^{2}p^{2}k}\right))-(1-\exp(-1.5c))^{k}-\mathcal{O}\left({n^{-1}}\right)$,444More precisely, we assume $(1-\delta)p^{2}k\geq c_{0}^{-1}\delta^{-2}\log k$ for a fixed constant $c_{0}>0$. Then with probability at least $1-6e^{-c_{0}\delta^{2}(1-\delta)pk}-(1-\exp(-1.5c))^{k}-\mathcal{O}\left({n^{-1}}\right)$, the statement in the theorem holds. as long as

[TABLE]

for all node $j\in K^{c}$ adjacent to $K$, where $C>0$ is a universal constant.

We require both $\alpha\in[0.1,0.9]$ and $\delta\leq 0.1$ in the theorem to avoid overly complicated constants; while this simplifies the presentation of the theorem, it is not difficult to show that a similar result holds more generally. Importantly, when combined with Theorem 7 (full recovery of the cluster), our result Theorem 9 immediately establishes that $\ell_{1}$-regularized PageRank recovers the target cluster exactly, even when the target cluster is constant-sized. We state this result informally in the following, which requires no proof.

Corollary 10** (Exact recovery; informal statement.)**

Under the same assumptions as in Theorem 7 and Theorem 9, there is a good starting node in $K$ such that $\ell_{1}$-regularized PageRank parameterized with that node as a seed node satisfies

[TABLE]

with nonvanishing probability.

It should be noted that a sort of condition like (8) about the realized degree seems necessary in order that the $\ell_{1}$-regularized PageRank has no false positives. The optimization program (2) assigns less weights to low degree nodes in the $\ell_{1}$ penalty, so any nodes adjacent to $K$ will become active unless the $\ell_{1}$-regularized PageRank penalizes them with nontrivial weights. Unlike Theorem 7 and Theorem 8, condition (8) rules out some specific models to which Theorem 9 can be applied. For example, planted clustering model with $p=\mathcal{O}\left({1}\right)$ and $q=\mathcal{O}\left({1/n}\right)$ does not satisfy this condition because the degrees in $K^{c}$ do not concentrate. For the stochastic block model, this condition is still satisfied if nodes adjacent to the target cluster belong to the clusters with degree larger than $\mathcal{O}\left({1/\gamma p}\right)=\mathcal{O}\left({1}\right)$. In practice, condition (8) may not be always applicable for every node adjacent to $K$, in which case the nodes that violate this condition may enter the model as false positives. We require the condition here though, since our model is essentially local and we do not have control outside $K$ beyond its neighbors.

3.3 Comparison with existing results

The local graph clustering problem has been relatively well-studied in the area of theoretical computer science, and the existing works largely focus on the worst-case guarantees. We now compare our results through the random graph model with the current known state-of-the-art worst-case results, given by Zhu et al. (2013).

First, the main result Theorem 1 of Zhu et al. (2013), when applied to our population graph $\overline{G}$, implies that $\textnormal{Vol}(\text{FP}),\textnormal{Vol}(\text{FN})\leq\textnormal{Vol}(K)\cdot\mathcal{O}\left({(1-\gamma)\log k}\right)$, as long as $\text{Gap}=\mathcal{O}\left({1/((1-\gamma)\log k)}\right)\geq\mathcal{O}\left({1}\right)$. When $\gamma=\mathcal{O}\left({1}\right)\in(0,1)$, our Theorem 7 states that if $pk^{2}\geq\mathcal{O}\left({\log k}\right)$, the output of the $\ell_{1}$-regularized PageRank model does not contain any false negative. This cannot be deduced from Zhu et al. (2013). In addition, our general bound on false positive, i.e., $\textnormal{Vol}(\text{FP})\leq\textnormal{Vol}(K)\cdot(\mathcal{O}\left({1/\gamma^{2}}\right)-1)$ in Theorem 8, is better than the worst-case bound of Zhu et al. (2013) in the regime of large $\gamma$, which is typically the case for many interesting scenarios. For instance, when the expected target conductance $\overline{\text{Cond}}=1-\gamma$ is small and fixed, the bound of the worst-case result degrades as the size of the target cluster $k$ increases, whereas our result is improved by increasing the probability bound. In the regime of $p=\mathcal{O}\left({1}\right),q=\mathcal{O}\left({1/n}\right)$, and $k=\mathcal{O}\left({1}\right)$ (hence $\gamma=\mathcal{O}\left({1}\right)$), our Theorem 9 shows that the output even contains no false positive. In this particular case, the strong separability ($p=\mathcal{O}\left({1}\right),q=\mathcal{O}\left({1/n}\right)$) corresponds to a constant signal-to-noise ratio, since even for $q=\mathcal{O}\left({1/n}\right)$ there are still a constant amount of edges outgoing from the target cluster, while the internal edges inside the target cluster is also constant. Although in practice the exact recovery of the target cluster may be a strong requirement, nevertheless, for real world clusters with high signal-to-noise ratio, $\ell_{1}$-regularized PageRank can still reconstruct the ground truth clusters more or less exactly (see, for instance, Section 5.2).

Finally we briefly give a comparison of our theoretical results with the information theoretic results of Chen and Xu (2016) in the special case of planted clustering model. To ease and simplify the comparison, we only consider the case where $p=O(1)$ and $q=O(\log n/n)$. In this case, Chen and Xu (2016, Theorem 2.5) implies that a solution to a SDP achieves exact recovery as long as $k\geq O(\log n)$, whereas our Theorem 7 and 8 suggest that in the same regime $\ell_{1}$-regularized PageRank fully recovers the target cluster while picking up a constant proportion of false positives. Importantly, $\ell_{1}$-regularized PageRank is essentially a local method, while Chen and Xu (2016)’s SDP is a global method that explores the entire graph.

3.4 Approximate personalized PageRank on random graph

One advantage of variational formulation of local graph clustering, via $\ell_{1}$-regularized convex program (2), is that it allows tractable analysis of the method in random graphs. Given the connection between $\ell_{1}$-regularized PageRank and APPR established in Theorem 5, this further allows us to obtain the statistical guarantees of the APPR algorithm under the random graph model. We formally state this result in the following theorem, which is a simple consequence of Theorem 5, in combination with those obtained in Section 3.1 and 3.2. (We write $\text{FP}(x^{\text{APPR}})=\{i\in\operatorname{support}(x^{\text{APPR}}):i\in K^{c}\}$ to denote the collection of false positive nodes in the APPR’s output.)

Theorem 11** (Recovery guarantees for APPR.)**

Consider the APPR algorithm given in (3) with parameter $\rho=\rho(\delta)$. Under the same conditions as Theorem 7, the output vector $x^{\text{APPR}}(\rho(\delta))$ satisfies

[TABLE]

with nonvanishing probability. Furthermore, under the same conditions as Theorem 7, there is a good starting node in $K$ such that the APPR’s output vector parameterized with that node as a seed node satisfies

[TABLE]

with nonvanishing probability, as long as

[TABLE]

for all node $j\in K^{c}$ adjacent to $K$.

4 Stagewise PageRank and solution paths

The $\ell_{1}$-regularized PageRank problem (2) is convex and there are numerous ways to solve it using convex optimization techniques. In Fountoulakis et al. (2019), the authors apply proximal gradient descent (ISTA) and show that the algorithm enjoys the locality property, meaning that the algorithm touches at most the optimal support set and its neighbors, while inheriting the fast convergence property of the proximal gradient descent. Optimization algorithms typically solve the problem at a single value of $\rho$, or a sequence of multiple values. On the other hand, “path algorithms” are designed to solve the problem for all values of a regularization parameter $\rho\in(0,\infty]$, or a subset of it when terminated early. This is more desirable when we need solutions over a list of parameter values.

The idea of designing path algorithms has already gained much attention in the sparse regression literature, first pioneered by Efron et al. (2004) and further developed by Zou et al. (2007); Hastie et al. (2004); Arnold and Tibshirani (2016). This has rendered the exploration of full regression coefficient paths relatively inexpensive. Unlike regression setting, however, this type of path algorithm has been less studied in local graph clustering (Gleich and Kloster, 2016). Motivated by the path algorithms developed in the sparse regression setting, we consider the following coordinate-wise algorithm for local graph clustering:

• •

Initialize $x^{(0)}=0$;

• •

For each $k\geq 0$, iterate the steps

[TABLE]

• •

Output a sequence $\{x^{(0)},x^{(1)},\ldots\}$.

The two main features of this algorithm are: 1) we greedily select the coordinate $i$ at each iteration that maximizes the magnitude of gradient, and 2) we update the current iterate by adding a small step size $\eta$ to the $i$th coordinate. This conservative update of the variable counterbalances the greedy selection step, thus making the algorithm more stable.

The above algorithm is called the “stagewise” algorithm, and in fact it has been widely studied by many authors (Efron et al., 2004; Hastie et al., 2007; Rosset et al., 2004; Rosset and Zhu, 2007; Zhao and Yu, 2007; Tibshirani, 2015). The stagewise algorithm is known to have implicit regularization effect closely related to $\ell_{1}$ norm regularization (Tibshirani, 2015), and in particular, if each component of the $\ell_{1}$-regularized solution has a monotone path, then the stagewise path exactly coincides with that of $\ell_{1}$ regularization, for $\eta\to 0$ (Efron et al., 2004; Rosset et al., 2004). Recalling our earlier result on the monotone path property of $\ell_{1}$-regularized PageRank (Lemma 4), the following result is then immediate.

Corollary 12

The output sequence of stagewise algorithm described in (9) converges to the $\ell_{1}$-regularized PageRank solution path as the step size goes to [math], i.e., $\eta\to 0$.

Therefore, the stagewise algorithm allows us to provably explore the entire $\ell_{1}$-regularization path via a single run of simple iterative steps. Another advantage of the stagewise algorithm is that it enjoys the locality property, in that the algorithm only touches the chosen nodes and its neighbors as it progresses. This is obvious from the update step of (9) and the expression of the gradient $\nabla_{i}f(x)$. Thus, when terminated early, the algorithm produces an approximate and partial solution path without making access to the entire graph.

For the $\ell_{1}$-regularized PageRank method, the parameter $\rho$ controls the extent to which the random walk has moved farther from the seed, so different values of $\rho$ reveal various scales of local clustering structure around the seed node. Therefore, in the setting of local graph clustering, the stagewise algorithm allows us to provably and efficiently track the evolution of a $\ell_{1}$-regularized PageRank diffusion and better understand the local cluster properties of the graph. This is well-suited for the purpose of exploratory graph analysis, and the idea of using path algorithms for exploring the graph has been also studied in Gleich and Kloster (2016). In addition to the exploratory analysis, the stagewise algorithm can be a competitive algorithm to find the target cluster if the size of the target cluster is small and/or medium and one needs a fine scale resolution of the solution path. However, when the size of the target cluster is quite large, using optimization algorithms with a coarse grid of regularization parameter may lead to better computational savings without exploring the entire solution path from scratch. Overall, the stagewise algorithm must be used in a complementary way to the optimization algorithms that directly solve (2). We also refer the readers to Tibshirani (2015) for comprehensive study of the stagewise algorithm for general sparse modeling problem.

In Figure 1, we compare the actual solution path to the stagewise algorithm paths for different step sizes. We generate data from the stochastic block model with $p=0.5,q=0.002$ and $r=50$. Each cluster has $20$ nodes. Figure 1 shows the $\ell_{1}$-regularization path and stagewise component paths for one particular draw from the stochastic block model. Here we only show the solution paths for nodes in the target cluster without seed node, among $n=1000$ nodes (each color line corresponds to different nodes in the target cluster). Note that when the step size is small, $\eta=0.0001$, the stagewise path appears to closely match to the $\ell_{1}$-regularization path; for moderate step size, $\eta=0.0005$, the stagewise path exhibits some jagged pattern but nevertheless accurately approximates the optimal path; and for relatively large step size, $\eta=0.001$, while the jagged pattern becomes more evident visually, it is clear that the overall trend still coincides well with that of the $\ell_{1}$ solution path.

5 Numerical evaluation

In this section, we provide a detailed numerical evaluation, to illustrate the performance of $\ell_{1}$-regularized PageRank on synthetic and real data. We have conducted a comprehensive experiments to demonstrate the state-of-the-art performance of the method across a wide range of real graphs, which has not been investigated at this level of extent in the prior works. To measure the quality of the recovered cluster, we define Precision and Recall as $\textnormal{Vol}(\mbox{TP})/\textnormal{Vol}(\operatorname{support}(\widehat{x}))$ and $\textnormal{Vol}(\mbox{TP})/\textnormal{Vol}(K)$, respectively. The F1score is the harmonic mean of precision and recall, $\left(2/\left(\text{Precision}^{-1}+\text{Recall}^{-1}\right)\right)$. We will also make use of conductance, where recall $\text{Cond}=\mbox{Cut}(S,S^{c})/\min\left(\textnormal{Vol}(S),\textnormal{Vol}(S^{c})\right)$. The lower the conductance value is, the better the quality of the cluster is. For our experiments, we solve problem (2) using a proximal coordinate descent algorithm, which enjoys both the “strongly local” running time property (running time depends on the size of cluster rather than the entire graph) as well as linear convergence (Fountoulakis et al., 2019).

5.1 Simulated data

In Figure 2, we demonstrate the performance of $\ell_{1}$-regularized PageRank as $\gamma$ increases (where, recall, $\gamma$ is defined in Eqn. (5)). We illustrate the performance of $\ell_{1}$-regularized PageRank in two cases. The first (red line with diamonds, “Best performance”) is a best-case scenario where we select the solution with the best F1score out of all the solutions that are produced by the stage-wise algorithm (9). The second scenario (blue line with squares, “Perf. for minimum conductance”) shows a more realistic case where we select the solution with minimum conductance out of all the solutions that are produced by the stage-wise algorithm. We note that F1score for the best-case scenario scales linearly as a function of $\gamma$. On the other hand, the scenario where we select the solution with minimum conductance scales sub-linearly as a function of $\gamma$ until a phase-transition around $\gamma\approx 0.75$, after which the scenario of minimum conductance matches the best-case scenario. The gap between the best-case scenario and the minimum conductance scenario indicates the need for new post-processing methods beyond minimum conductance, which we leave for future work. For the experiments in Figure 2, we fix the teleportation parameter $\alpha=0.1$. We generate graphs from the stochastic block model which consists of $10$ clusters, each of which has $20$ nodes, and only one of which is the target cluster $K$. We use the same parameters $p$ and $q$ across different clusters to generate edges within and between clusters. Here we set $p=0.5$ and $q$ is varying in order to generate various $\gamma$ as is shown in Figure 2. The results are averaged over $30$ trials.

In Figure 3, we demonstrate a much more detailed view of the performance of $\ell_{1}$-regularized PageRank for four representative values of $\gamma$. More precisely, we show that for large and medium values of $\gamma$, there exists a parameter $\rho$ such that the support of the $\ell_{1}$-regularized PageRank solution recovers the target cluster; and, for smaller $\gamma<0.5$, $\ell_{1}$-regularized PageRank does not recover the target cluster with high accuracy. For Figure 3 we use the same experiment setting as in Figure 2. For large $\gamma$, Figure 3(a), we observe that when $\ell_{1}$-regularized PageRank recovers about $20$ nodes, then these nodes correspond to very high precision and recall, and as the number of nodes in the solution increases then precision decreases. We also observe that conductance of the recovered cluster is a good metric for finding the target cluster. By this, we mean that we will find the target cluster with high precision and recall if out of all solutions on the path we choose the one with minimum conductance. As $\gamma$ gets smaller, e.g., in Figure 3(c), the minimum conductance does not relate to the target cluster. However, it is clear from Figures 3(b) and 3(c) that $\ell_{1}$-regularized PageRank with minimum conductance still finds the target cluster $K$ with good accuracy if the algorithm is terminated early. Finally, in Figure 3(d), we demonstrate a case where $\gamma$ is small and conductance of solution fails to relate to the target cluster and there is no output of $\ell_{1}$-regularized PageRank that recovers the target cluster accurately.

5.2 Real data experiments

Next, we test the performance of local graph clustering methods using biology networks and social networks. All the graphs that we consider are unweighted and undirected. For a summary of the datasets see Table 1. All datasets that are used come with suggested ground-truth clusters for social networks and with a gold standard for biology networks. However, we filter the given ground-truth clusters by measuring their conductance values. In particular, we keep only the ground-truth clusters that have conductance value less than or equal to $0.6$. This means we keep the clusters that the number of edges that cross the cluster is $60\%$ of the volume of the cluster. This way we obtain a wide range of clusters from “good” (small conductance) to noisy (large conductance, i.e., close to $0.6$).

5.2.1 Datasets

Sfld. This dataset contains pairwise similarities of blasted sequences of $232$ proteins belonging to the amidohydrolase superfamily (Brown et al., 2006). There are $232$ nodes and $31140$ edges in this graph. A gold standard is provided describing families within the given superfamily. According to the gold standard the amidrohydrolase superfamily contains $29$ families/clusters. However, after filtering the families we find that only two have conductance value less than $0.6$. (This should not be surprising, given that the graph is so dense.) In Table 2 we present properties of the two clusters that correspond to urease.0 and AMP amidrohydrolase superfamilies.

PPI-mips. This dataset is a protein-protein interaction graph of mammalian species (Pagel et al., 2004). There are $1096$ nodes and $26442$ edges in this graph. In Table 2 we provide all suggested ground-truth clusters that have conductance less than $0.6$.

FB-Johns55. This graph is a Facebook anonymized dataset on a particular day in September $2005$ for a student social network at John Hopkins university. The graph is unweighted and it represents “friendship” ties. The data form a subset of the Facebook100 data-set from Traud et al. (2011, 2012). This graph has $5157$ nodes and $186572$ edges. This dataset comes along with $6$ features, i.e., second major, high school, gender, dorm, major index and year. We construct “ground truth” clusters by using the features for each node. In particular, we consider nodes with the same value of a feature to be a cluster, e.g., students of year $2009$. As analyzed in the original paper that introduced these datasets (Traud et al., 2012), for FB-Johns55 the year and major index features give non-trivial “assortativity coefficients”, which is a “local measure of homophily”. This agrees with the ground truth clusters we find after applying our filtering technique. The clusters per graph are shown in Table 3.

Colgate88. This graph is constructed similarly to the FB-Johns55 graph but for Colgate University. We apply the same filtering techniques as for FB-Johns55 and we present the filtered clusters in Table 3. There are $3482$ nodes and $155043$ edges in this graph.

Orkut. Orkut is a free on-line social network where users form friendship each other. Orkut also allows users form a group which other members can then join. This dataset has $3072441$ nodes and $117185083$ edges. It can be downloaded from Leskovec and Krevl (2014). This dataset comes with $5000$ ground truth communities, which we filter by maintaining the clusters with conductance less than or equal to $0.6$. Out of the $5000$ communities only $282$ pass our filtering test.

The above real graphs may not be generated from the random model that we consider in Section 3, nonetheless, we have evaluated the method to illustrate the scenarios that are not just idealized. Moreover, the Facebook social networks and the biology networks are expected to exhibit homophily structure for some ground truth clusters that are highly inter-connected relative to the rest of the graph. These block-structured networks belong to a special class of networks and we believe that random graph models, such as stochastic block model, can closely approximate it.

5.2.2 Methods

We compare the performance of $\ell_{1}$-regularized PageRank with state-of-the-art local graph clustering algorithms. There are two categories of local graph clustering methods: local spectral (Andersen et al., 2006) and it’s follow-up work (Zhu et al., 2013); and local flow-based methods (Lang and Rao, 2004; Andersen and Lang, 2008; Orecchia and Zhu, 2014; Veldt et al., 2016).

Approximate personalized PageRank (APPR): In Andersen et al. (2006) and Zhu et al. (2013) an approximate personalized PageRank (APPR) algorithm is proposed, where the personalized PageRank linear system is solved approximately using a local diffusion process. Both papers study nearly identical algorithms, with the latter paper giving better theoretical guarantees based on a stronger assumption. In particular, the authors in Zhu et al. (2013) assume that the internal connectivity of the target cluster (minimum conductance of the induced subgraph for the target cluster) is quadratically stronger than the conductance of the target cluster.

SimpleLocal (SL): Out of the four flow-based methods Lang and Rao (2004); Andersen and Lang (2008); Orecchia and Zhu (2014); Veldt et al. (2016), the one proposed in Veldt et al. (2016) (SimpleLocal) simplifies and generalizes the methods proposed in Lang and Rao (2004); Andersen and Lang (2008); Orecchia and Zhu (2014), while having similar theoretical and practical guarantees in terms of quality of the output. Depending on the parameter tuning of SimpleLocal, one can obtain any of the flow-based methods proposed in Lang and Rao (2004); Andersen and Lang (2008); Orecchia and Zhu (2014). Therefore, we will only use SimpleLocal in our experiments, and we will use different parameter tuning such that we obtain performance for a wide-range of methods. Moreover, SimpleLocal requires initialization with a set of seed nodes that ideally has some overlap with the target cluster. Therefore, in this paper, we will use SimpleLocal with three different initialization techniques. Below we comment on why we choose the following inputs to SimpleLocal.

$\ell_{1}$-reg. PR-SL: SimpleLocal using the output of $\ell_{1}$-reg. PR as input. 2. 2.

BFS-SL: We will initialize SimpleLocal using the output of a breadth-first-search-type (BFS) algorithm starting from a given seed node. The algorithm that is used for initialization of SimpleLocal is shown in Algorithm 1. Details about how we use this algorithm are given in the next subsection. Briefly, Algorithm 1 is used as a seed node expansion technique, which has also been used in Veldt et al. (2016).

Note that all these flow-based methods are “improve” methods, i.e., they are not stand-alone, and they require initial input from some other stand-alone method, such as APPR or $\ell_{1}$-reg. PR. This is both a theoretical and a practical argument. It is a theoretical argument because in the theoretical analysis of SimpleLocal (see Theorem 3 in Orecchia and Zhu (2014)), the input to SimpleLocal is required to have sufficient overlap with the target cluster. Otherwise, currently there is no guarantee that flow-improve methods output a reasonable approximation to the target cluster in terms of its conductance value (Theorem 1 in Orecchia and Zhu (2014)) or in terms of false and true positives (Theorem 3 in Orecchia and Zhu (2014)). Theorem 3 in Orecchia and Zhu (2014) suggests that sufficient overlap can be achieved either by using a local spectral method such as APPR or $\ell_{1}$-reg. PR. In this paper, we use only the latter. This is because, as shown in Fountoulakis et al. (2019), theoretically and empirically APPR is very similar to $\ell_{1}$-reg. PR. We also show extensive experiments in this section about the similarity of APPR and $\ell_{1}$-reg. PR. In practice one could use heuristics such as a BFS-type algorithm to slightly expand a seed node within a target cluster and then provide the output of the BFS-type as input to SimpleLocal, i.e., see also Veldt et al. (2016). In Algorithm 1 we provide a pseudo-code for the BFS-type algorithm that we use in this paper. Basically, the only difference with a standard BFS algorithm is that we explore neighbors in batches, which allows us to terminate the BFS algorithm after a given number of steps.

5.2.3 Parameter tuning

APPR and $\ell_{1}$-reg. PR have two parameters, the teleportation parameter $\alpha$ and a tolerance/$\ell_{1}$-regularization parameter $\rho$. The teleportation parameter should be set according to the reciprocal of the mixing time of a random walk within the target cluster, which is equal to the smallest nonzero eigenvalue of the normalized Laplacian for the subgraph that corresponds to the target cluster. See Zhu et al. (2013) for details. Let us denote the eigenvalue with $\lambda$. In our case the target cluster is a given ground truth cluster. We use this information to set parameter $\alpha$. In particular, for each node in the target cluster we run APPR and $\ell_{1}$-reg. PR $4$ times where $\alpha$ is set based on a range of values in $[\lambda/2,2\lambda]$ with a step of $(2\lambda-\lambda/2)/4$. The regularization parameter of $\ell_{1}$-reg. PR has nearly identical purpose as the tolerance parameter of APPR. Both parameters are set to be proportional to inverse of the volume of the target cluster, as suggested by theoretical arguments in this paper as well as in Andersen et al. (2006); Zhu et al. (2013); Fountoulakis et al. (2019). For each parameter setting we use sweep cut to round the output of APPR and $\ell_{1}$-reg. PR to find a cluster of low conductance. The sweep cut rounding procedure is a common technique to post-process the output of local graph clustering methods—for details, we refer the reader to Andersen et al. (2006); Zhu et al. (2013); Fountoulakis et al. (2019). We use a proximal coordinate descent algorithm (Fountoulakis et al., 2019) to solve the $\ell_{1}$-reg. PR problem. Over all parameter settings, we return the cluster with the lowest conductance value as an output of APPR and $\ell_{1}$-reg. PR.

SimpleLocal has only one parameter denoted by $\delta$. The $\delta$ parameter controls how localized the output is going to be around the input seed nodes (Veldt et al., 2016), it also controls the quality of the output in terms of its conductance value (Theorem 1 in Orecchia and Zhu (2014)) or in terms of false and true positives (Theorem 3 in Orecchia and Zhu (2014)). The parameter has to satisfy $\delta\geq 0$, and according to Orecchia and Zhu (2014) it should be set such that $\textnormal{Vol}(R)/\textnormal{Vol}(V-R)+\delta=1/(3/\sigma+3)$, where $R$ is the set of seed nodes that is given as input to SimpleLocal, and $\sigma$ is a parameter that satisfies $\sigma\in[\textnormal{Vol}(R)/\textnormal{Vol}(V-R),1]$. It is suggested in Orecchia and Zhu (2014) to set $\sigma=\mathcal{O}(\textnormal{Vol}(R\cap K)/\textnormal{Vol}(K))$, where $K$ denotes the target cluster. Therefore, in our experiments we set

[TABLE]

When we use $\ell_{1}$-reg. PR as input to SimpleLocal, this parameter setting also provides the theoretical guarantees which are claimed in Orecchia and Zhu (2014) (assuming that all assumptions about the target cluster are satisfied).

We set parameter steps $=n$ in Algorithm 1 and we terminate Algorithm 1 when the current set of seed nodes, denoted by seeds, satisfies $\textnormal{Vol}(\mbox{{seeds}}\cap K)/\textnormal{Vol}(K)\geq 0.75$, i.e., the output of Algorithm 1 has to have at least $75\%$ overlap with the target cluster, or when $\textnormal{Vol}(\mbox{{seeds}})/\textnormal{Vol}(G)\geq 0.25$, i.e., the volume of the output is larger than $25\%$ the volume of the graph. This setting is also motivated by Theorem 3 in Orecchia and Zhu (2014), which mentions that the input to SimpleLocal has $75\%$ overlap with the target cluster. However, since this relies on the assumption that the input set of nodes is the output of APPR or $\ell_{1}$-reg. PR, which are themselves clustering algorithms, while Algorithm 1 is not, then we also add the second termination criterion that the output cannot have volume more than $25\%$ the volume of the graph.

5.2.4 Results

APPR and $\ell_{1}$-reg. PR are similar. We start our empirical observations by comparing APPR and $\ell_{1}$-reg. PR. We already proved in Theorem 5 that these methods are similar. However, we would like to point out some additional details, which will help us justify simplification of the experiments for the remainder of this section. In Figure 4, we make a much more extended empirical comparison between the two local spectral methods, where we compare their precision, recall and F1score. We use the Orkut dataset, and we compare the two methods using all $282$ ground truth clusters of this dataset. In this figure, we present average results over all nodes for each given ground truth cluster. We observe that APPR and $\ell_{1}$-reg. PR produce output with nearly identical precision, recall and F1score. We attribute any minor differences between the two methods to the minor differences between the optimality conditions of $\ell_{1}$-reg. PR and APPR. Moreover, APPR and proximal coordinate descent for $\ell_{1}$-reg. PR have similar running time in practice for producing the results in Figure 4. This is justified by worst-case analysis in Fountoulakis et al. (2019) as well as our average-case analysis in Theorem 11. In particular, each method required $75$ hours to produce Figure 4, which required $304960$ calls to APPR and $\ell_{1}$-reg. PR. Due to the similarity of the two approaches, we will only use $\ell_{1}$-reg. PR in the subsequent experiments. Note that in this experiment we do not show performance of flow-based methods because based on our empirical observations on small datasets like FB-Johns55 and Colgate88 it would have taken close to a month to obtain similar plots like in Figure 4. (Improving this situation for local flow improve methods is an important direction for future work.) For example, see the running times of flow-based methods for FB-Johns55 and Colgate88 in Table 7.

APPR and $\ell_{1}$-reg. PR work well for low conductance target clusters. Here, we comment on the performance of APPR and $\ell_{1}$-reg. PR in Figure 4 on the Orkut dataset. We observe that performance of both methods decreases as the conductance of the target cluster increases, i.e., as the cluster quality gets worse. This is an expected outcome that is predicted by the average-case analysis of this paper. However, it is important to mention that, for target clusters with conductance close to $0.4$, the F1score of $\ell_{1}$-reg. PR is around $0.8$. This is surprisingly good performance since a cluster with conductance $0.4$ means that roughly half of the edges of the target cluster cross the cluster boundary (which implies that such target clusters are actually quite noisy and not of particularly high quality).

Results for biology datasets and Sfld and PPI-mips. The results for the biology datasets are shown in Table 4. We present average results over all nodes for each given ground truth cluster. We denote with bold numbers the performance number of a method when it has the largest F1score among all methods.

We note the consistent state-of-the-art performance of $\ell_{1}$-reg. PR for all clusters in Table 4. For some ground truth clusters, $\ell_{1}$-regularized PageRank perfectly recovers the target clusters, which is mainly attributed to the fact that the ground truth clusters have strong separability property (see also Corollary 10 of the main text). In most experiments, SimpleLocal did not improve the performance of the input of $\ell_{1}$-reg. PR, but also it did not make it worse. In some cases, like the Spindle ground truth cluster in the PPI-mips dataset, SimpleLocal decreased the performance of $\ell_{1}$-reg. PR in terms of the F1score. This is because SimpleLocal found clusters that have smaller conductance, but that do not correspond to clusters with the highest F1score. This is a known issue that has been mentioned in Fountoulakis et al. (2017). BFS-SL has the worst performance among all methods in most experiments. In fact, BFS-SL performs well only for the usrease ground truth cluster in the Sfld dataset. The performance of BFS-SL is especially poor for all ground truth clusters in the PPI-mips dataset. It is important to mention that we did experiment with different parameter tuning for both BFS-type Algorithm 1 and SimpleLocal for the BFS-SL method, but the performance was poor for all settings of parameters that we tried. We attribute the poor performance of BFS-SL in the BFS-type algorithm, which provides the input to SimpleLocal. In particular, the BFS-type Algorithm 1 is not related to clustering in a general sense, and this translates to poor quality input to SimpleLocal. As is mentioned in the theoretical analysis in Orecchia and Zhu (2014); Veldt et al. (2016), SimpleLocal requires as input the output of a local spectral method such as $\ell_{1}$-reg. PR in order to perform well, which is also verified by the results in Table 4.

The corresponding running times for each method are shown in Table 5. Each numeric value in this table demonstrates the total running time to run a method for a given ground truth cluster. The running time is the sum of the running times for each node in the ground truth cluster. Note that $\ell_{1}$-reg. PR is the fastest method.

Results social networks FB-Johns55 and Colgate88. The results for the social network graphs are shown in Table 6. There are a lot of interesting observation for this set of experiments. First, $\ell_{1}$-reg. PR outperforms BFS-LS, with the exception of the ground truth clusters of major index 217 in FB-Johns55, where BFS-LS has a $0.03$ larger F1score, and the ground truth cluster of year 2009 in Colgate88, where BFS-SL has the same F1score as $\ell_{1}$-reg. PR. We observed two reasons that BFS-SL has worse performance in most experiments. The first reason is that BFS-SL outputs a cluster that has smaller conductance than the output cluster of $\ell_{1}$-reg. PR, but better conductance is often not related to the ground truth cluster, especially in cases where the ground truth cluster itself has large conductance. We attribute this behavior to SL because as an algorithm it attempts to find a cluster with small conductance. The second reason is that the input to SL from BFS-type Algorithm 1 is not a good approximation to the ground truth cluster, which is a required property of SL such that it performs well.

The second set of observations is about $\ell_{1}$-reg. PR-SL. For most ground truth clusters, we observe that $\ell_{1}$-reg. PR-SL performs worse than or on par with $\ell_{1}$-reg. PR, with the exception of ground truth clusters year 2009 and major index 217 in FB-Johns55 and ground truth clusters of years 2008 and 2009 in Colgate88. When $\ell_{1}$-reg. PR-SL makes the input of $\ell_{1}$-reg. PR worse, it is clearly because the former finds a cluster with better conductance value which does not relate to the ground truth cluster. We observe this behavior very often when the target cluster does not have small conductance value, and this is also confirmed through our simulation study (see Figure 3(d) in Section 5.1). When $\ell_{1}$-reg. PR-SL performs better it is because the ground truth cluster has small conductance but not small enough such that $\ell_{1}$-reg. PR performs well by itself. In particular, $\ell_{1}$-reg. PR leaks more mass outside of the target cluster than it should, and this results in small precision. This is a well-known problem that has been also observed in Fountoulakis et al. (2017), which can be fixed by SL.

The corresponding running times for each method are shown in Table 7. The running time is the sum of the running times for each node in the ground truth cluster. Note that $\ell_{1}$-reg. PR is the fastest method.

Memory scalability for small- and large-scale graphs. One of the main motivations of the $\ell_{1}$-regularized PageRank model is the low memory requirements for each seed node. In Table 8 we illustrate performance in terms of memory requirements for solving the $\ell_{1}$-regularized PageRank using proximal coordinate descent. In particular, we illustrate how memory requirement increases as $\rho$ decreases. As is expected, for large values of $\rho$ the output solution is very sparse and the algorithms have very low memory requirements. As $\rho$ gets smaller then the solutions become denser and memory requirements increase. We observe similar performance for FB-Johns55 (small dataset) and Orkut (large dataset).

6 Conclusion

In this paper, we have examined the $\ell_{1}$-regularized PageRank optimization problem for local graph clustering. In a local graph clustering problem, the objective is to find a single target cluster in a large graph, given a seed node in the cluster, and to do so in a running time that does not depend on the size of the entire graph, but instead that depends on the size of the output cluster. Algorithmic results (i.e., running-time bounds to achieve a given cluster quality) for local graph clustering abound, but our results are the first statistical results (i.e., where one is interested in recovering a cluster under an hypothesized model). Under our local random model, we show that the optimal support of $\ell_{1}$-regularized PageRank identifies the target cluster with bounded false positives, and in certain settings exact recovery is also possible. We further establish a strong connection between $\ell_{1}$-regularized PageRank and approximate personalized PageRank (APPR), based on which we obtain similar statistical guarantees on APPR under the random model. Additionally, we have brought the idea of solution path algorithms from the sparse regression literature to the local graph clustering literature, and we showed that the forward stagewise algorithm gives a provable approximation to the entire $\ell_{1}$ regularization path of this algorithm. Finally we demonstrate the state-of-the-art performance of $\ell_{1}$-regularized PageRank on both simulated and real data graphs.

Acknowledgments

We would like to acknowledge support for this project from the National Science Foundation (NSF grant IIS-9988642) and the Multidisciplinary Research Program of the Department of Defense (MURI N00014-00-1-0637). This material is also based in part on research sponsored by DARPA and the Air Force Research Laboratory under agreement number FA8750-17-2-0122. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of DARPA and the Air Force Research Laboratory or the U.S. Government. We would also like to acknowledge ARO, NSF, ONR, and Berkeley Institute for Data Science for providing partial support of this work.

A Some auxiliary lemmas

To establish the theorems, we need a few concentration inequalities and intermediate results that shall be used in the proof. In Section A.1, we give degree concentration inequalities for random graphs, and in Section A.2, we state recovery guarantees of $\ell_{1}$-regularized PageRank on the population graph. Section A.3 gives a few important results on the $\ell_{1}$-regularized PageRank when restricted to the target cluster.

A.1 Concentration lemmas

Here, we present several concentration lemmas for degrees of random graphs. The first lemma is consequence of Chernoff’s inequality for the sum of independent Bernoulli random variables, and applying union bound (c.f. see Vershynin (2018, Theorem 2.3.1)).

Lemma 13** (Adapted from Vershynin (2018, Proposition 2.4.1).)**

Let $X_{i}$ be the sum of independent Bernoulli random variables, with $\mathbb{E}\left[{X_{i}}\right]=\mu$ for $i=1,2,\ldots,k$. Then, there exists a universal constant $c_{0}>0$ such that if $\mu\geq c_{0}^{-1}\delta^{-2}\log k$, then with probability at least $1-2e^{-c_{0}\delta^{2}\mu}$,

[TABLE]

According to our random model (Definition 6), the degree vector $d_{K}$ for the target cluster $K$ comprises of random variables which are the sum of independent Bernoulli random variables with common mean $\bar{d}$. Therefore, by applying Lemma 13, it is straightforward to see the following result.

Lemma 14** (Adapted from Vershynin (2018, Proposition 2.4.1).)**

There exists a universal constant $c_{0}>0$ such that if $\bar{d}\geq c_{0}^{-1}\delta^{-2}\log k$, then with probability at least $1-2e^{-c_{0}\delta^{2}\bar{d}}$,

[TABLE]

where $\bar{d}$ is the average degree of the vertices in the cluster $K$.

The next lemma bounds the number of edges between node $j\in K^{c}$ and the target cluster $K$ when $q=\mathcal{O}\left({\frac{1}{n}}\right)$.

Lemma 15

Suppose that $q=c/n$ and that $k\geq 2(c+3)$ for a positive constant $c$. Then for $n$ sufficiently large, with probability at least $1-\mathcal{O}\left({n^{-1}}\right)$,

[TABLE]

A.2 Exact recovery of target cluster in population graph

Next, we define the “ground truth” PageRank vector, which we obtain by applying $\ell_{1}$-regularized PageRank to the population graph $\overline{G}$ of our random model. Recall the adjacency matrix $\mathbb{E}\left[{A}\right]$ defined in (4), the associated diagonal degree matrix $\mathbb{E}\left[{D}\right]$, and the Laplacian matrix $\mathbb{E}\left[{L}\right]$. Writing $\mathbb{E}\left[{Q}\right]=\alpha\mathbb{E}\left[{D}\right]+\frac{1-\alpha}{2}\mathbb{E}\left[{L}\right]$, we denote the population version of $\ell_{1}$-regularized PageRank as

[TABLE]

Compared to (2), we see that the matrices associated with the graph are now replaced by their expected counterparts.

The following lemma shows that there exist a range of $\rho$ values for which the solution to the population version of the $\ell_{1}$-regularized PageRank minimization problem (10) gives an exact recovery for the target cluster.

Lemma 16** (Exact recovery for population PageRank vector.)**

Consider the local random model given in Definition 6 and suppose that Assumption 1 holds. Then, for $n$ sufficiently large, the “ground truth” PageRank vector $x^{\star}$, defined in (10), identifies $K$ correctly, i.e.,

[TABLE]

as long as $\rho\in[\rho^{\sharp},\rho^{\natural})$, where

[TABLE]

where $\mathbb{E}\left[{{d}_{\ell_{\sharp}}}\right]=\min_{\ell\in K^{c}}\mathbb{E}\left[{{d}_{\ell}}\right]$. Furthermore, $x^{\star}$ has a closed form expression, $x^{\star}=u\cdot\mathbf{1}_{S}+v\cdot\mathbf{1}_{K}$, where $u$ and $v$ are given by

[TABLE]

From the closed form expression, we can see that $x^{\star}$ has high probability mass on the single seed node with a long tail further away from the seed node. The mass outside the target cluster is being thresholded exactly to zero via $\ell_{1}$-norm regularization, thus identifying the target cluster without any false positives. Of course, in practice, the population graph is unknown, and we are instead given an instance of the graph from the random model. In this case, the ground truth vector $x^{\star}$ allows us to estimate the magnitude of the $\ell_{1}$-regularized PageRank vector $\widehat{x}$ by analyzing the error between $x^{\star}$ and $\widehat{x}$. For more details, we refer the reader to the proof of Lemma 19.

A.3 $\ell_{1}$-regularized PageRank restricted to target cluster

Consider the $\ell_{1}$-regularized PageRank problem restricted to the target cluster $K$.

[TABLE]

Since our aim is to recover the target cluster $K$ based on the local model (Definition 6), it is natural to analyze the properties of the solution to this reduced problem. Here, abusing notation, we use $\widehat{x}^{\textnormal{R}}$ to denote the vector either in the original space $\mathbb{R}^{n}$ or in the reduced space $\mathbb{R}^{k}$.

We present several lemmas about $\widehat{x}^{\textnormal{R}}$ that will be critical for the proof of theorems. First, we give a guarantee on the recovery of the target cluster for the optimal solution (A.3).

Lemma 17

Let $\widehat{x}^{\textnormal{R}}$ be the solution to the reduced problem (A.3). Suppose that $(1-\delta)p^{2}k\geq c_{0}^{-1}\delta^{-2}\log k$. Then, if the regularization parameter satisfies $\rho\leq\rho(\delta)$, we have

[TABLE]

with probability at least $1-6e^{-c_{0}\delta^{2}(1-\delta)p^{2}k}$. Here $c_{0}$ is the same constant appearing in Lemma 13.

For the above result, since by construction $\widehat{x}^{\textnormal{R}}$ is zero outside $K$, all we need to show is that $\widehat{x}^{\textnormal{R}}_{K}>0$ when $\rho\approx\mathcal{O}\left({\frac{\gamma p}{\bar{d}^{2}}}\right)$. Next we compare the support set of $\widehat{x}^{\textnormal{R}}$ to that of $\widehat{x}$.

Lemma 18

For any regularization parameter $\rho\in[0,\infty)$, we have

[TABLE]

Finally we have the following estimate on the maximum coordinate of $\widehat{x}^{\textnormal{R}}$ on $K\backslash S$ (recall $u$ and $v$ are respectively the components of $x^{\star}$ on $S$ and $K\backslash S$; see Lemma 16).

Lemma 19

Let $\delta\leq 0.1$ and suppose that $(1-\delta)p^{2}k\geq c_{0}^{-1}\delta^{-2}\log k$. Assume that Assumption 1 holds and that the size of the target cluster $k\geq 5$. If the regularization parameter satisfies $\rho\geq\rho(\delta)$, then for $n$ sufficiently large, the following bound holds

[TABLE]

with probability at least $1-6e^{-c_{0}\delta^{2}(1-\delta)p^{2}k}$. Here $c_{0}$ is the same constant appearing in Lemma 13 and $c_{1}$ is a positive numerical constant.

B Proofs of theorems

In this section we prove all of our theorems. Based on the lemmas presented above we give proofs of our five theorems, respectively, in Section B.1, B.2, B.3, B.4, and B.5.

B.1 Proof of Theorem 5

Fix $\rho_{0}>0$. First, we prove the first part of the theorem, i.e., $\operatorname{support}(\widehat{x}(\rho_{0}))\subseteq\operatorname{support}(x^{\text{APPR}}(\rho_{0}))$. When $\rho_{0}>1/d_{S}$, this relation holds trivially because both $\widehat{x}(\rho_{0})$ and $x^{\text{APPR}}(\rho_{0})$ do not contain any nodes in the support set. To prove that the relation holds for $\rho_{0}\leq 1/d_{S}$, we will proceed by induction. Specifically, we will prove that

[TABLE]

for all $\rho_{0}\in(0,1/d_{S}]$. Assuming that this holds, it is straightforward to follow that

[TABLE]

thus proving the claim.

Now we turn to proving (12). When $\rho_{0}=1/d_{S}$, we can see from the algorithm (3) that $x^{\text{APPR}}(\rho_{0})$ contains the seed node as the only element in the support set, while by Lemma 3, we have $\widehat{x}=0$, which proves (12). Next, assuming that it holds at some $\rho_{0}\in(0,1/d_{S}]$, let $\widetilde{\rho_{0}}>0$ be chosen such that $\widetilde{\rho_{0}}<\rho_{0}$ and such that there appears a node $i\in V$ that violates the condition (12) for the first time. If $i\in\operatorname{support}(x^{\text{APPR}}(\widetilde{\rho_{0}}))$, since the path $\widehat{x}(\rho)$ is continuous by the proof of Lemma 4, this means that $i$ is the first node such that $\widehat{x}_{i}(\widetilde{\rho_{0}})=x^{\text{APPR}}_{i}(\widetilde{\rho_{0}})>0$ (there may exist multiple nodes that simultaneously violate (12) at $\rho=\widetilde{\rho_{0}}$ but the same statement still applies). In this case, by induction, we know that $\widehat{x}_{j}(\widetilde{\rho_{0}})\leq x^{\text{APPR}}_{j}(\widetilde{\rho_{0}})$ for $j\neq i$. Then

[TABLE]

where the inequality holds since $(D^{-1}Ax)_{i}=\sum_{j\sim i}w_{ji}x_{j}/d_{i}$ for any $x\in\mathbb{R}^{|V|}$. By the optimality condition of (2) (Lemma 3), the left-hand side must be $(D^{-1}\nabla f(\widehat{x}(\widetilde{\rho_{0}})))_{i}=-\widetilde{\rho_{0}}\alpha$ which gives

[TABLE]

However, this is a contradiction to the termination criterion of the APPR algorithm (3), and in particular, we must have $i\notin\operatorname{support}(x^{\text{APPR}}(\widetilde{\rho_{0}}))$ and $\widehat{x}_{i}(\widetilde{\rho_{0}})>0$ by the condition (12). Then, by Lemma 3, this implies that $(D^{-1}\nabla f(\widehat{x}(\widetilde{\rho_{0}})))_{i}=-\widetilde{\rho_{0}}\alpha$, and following the same steps as above we can see that $-\widetilde{\rho_{0}}\alpha>(D^{-1}\nabla f(x^{\text{APPR}}(\widetilde{\rho_{0}})))_{i}$. However, this again contradicts the termination criterion of APPR, thus proving the first part of the theorem.

Next, we prove the second part of the theorem, i.e., $\operatorname{support}(x^{\text{APPR}}(\rho_{0}))\subseteq\operatorname{support}(\widehat{x}(\rho_{1}))$ for $\rho_{1}=(1-\alpha)\cdot\rho_{0}/2$. We first claim that for every node $i\in\operatorname{support}(x^{\text{APPR}}(\rho_{0}))$, we must have $-\rho\alpha d_{i}<\nabla_{i}f(x^{\text{APPR}}(\rho_{0}))\leq-\frac{1-\alpha}{2}\rho\alpha d_{i}$. The first inequality is obvious from the termination criterion of APPR. To see why the second inequality holds, if a node $i\in V$ is selected at iteration $k$ of the APPR algorithm, then by the update step (3), we have $x^{(k+1)}_{i}=x_{i}^{(k)}-d_{i}^{-1}\nabla_{i}f(x^{(k)})$. Then, the gradient of $f(x^{(k)})$ on node $i$ is updated as

[TABLE]

By the termination criterion of APPR, we know that $\nabla_{i}f(x^{(k)})\leq-\rho_{0}\alpha d_{i}$ for all $i\in V$. This shows that the right-hand side of the above equation is $\leq-\rho_{0}\alpha d_{i}$, and in particular, taking $k\to\infty$, we obtain $\nabla_{i}f(x^{\text{APPR}}(\rho_{0}))\leq-\frac{1-\alpha}{2}\rho_{0}\alpha d_{i}$, which proves the claim.

Now consider the following strategy of continuously adding small mass to the vector $x^{\text{APPR}}(\rho_{0})$:

Start with $x^{\text{APPR}}(\rho_{0})$. Find the node $i$ whose gradient is smallest, i.e.,

[TABLE] 2. 2.

Add mass to the node $j_{1}$ until some other node $j_{2}\in V$ has the gradient as small as $j_{1}$. This event always happens because by the work above (13) we can see that the gradient on node $j_{1}$ is increasing while the gradients on the other nodes are either decreasing (if it is a neighboring node of $j_{1}$) or stay the same. 3. 3.

Add mass to the nodes $(j_{1},j_{2})$ such that gradients on $j_{1}$ and $j_{2}$ are increasing at the same rate,555This is always possible since for any positive $\epsilon\in\mathbb{R}^{|V|}$, we know that $\sum_{i\in V}(\nabla_{i}f(x+\epsilon)-\nabla_{i}f(x))>0$. So, in order to increase the gradients at the same rate, we need to find a direction $\epsilon$ such that the summands $\nabla_{j}f(x+\epsilon)-\nabla_{j}f(x)$ are same for nodes $j$ that are selected. and until some other node $j_{3}\in V$ has the gradient as small as $j_{1}$ and $j_{2}$. 4. 4.

Continue this step until the gradient on the node $j_{1}$ becomes $-\frac{1-\alpha}{2}\rho_{0}\alpha d_{i}$.

Writing $\widetilde{x}^{\text{APPR}}$ to denote the output vector of the above procedure, it is easy to see that $\widetilde{x}^{\text{APPR}}$ indeed satisfies the optimality condition of the $\ell_{1}$-regularized PageRank minimization problem, given in Lemma 3, with $\rho=\rho_{1}$. This in turn shows that $\operatorname{support}(x^{\text{APPR}})\subseteq\operatorname{support}(\widehat{x}(\rho_{1}))$, since by the uniqueness of the optimal solution we have $\widetilde{x}^{\text{APPR}}=\widehat{x}(\rho_{1})$. This completes the proof of Theorem 5.

B.2 Proof of Theorem 7

The proof of Theorem 7 is a straightforward combination of Lemma 17 and 18. Specifically, by Lemma 17, we know that for $\rho\leq\rho(\delta)$, we have that $\operatorname{support}(\widehat{x}^{\textnormal{R}})=K$. Then applying Lemma 18 we have

[TABLE]

thus proving the result.

B.3 Proof of Theorem 8

To prove Theorem 8, we first need the following lemma for bounding the volume of $\operatorname{support}(\widehat{x})$. This lemma is a stronger version of Fountoulakis and Gondzio (2015, Theorem 2).

Lemma 20

For any regularization parameter $\rho>0$, it holds that

[TABLE]

where $\textnormal{Vol}(\operatorname{support}(\widehat{x}))=\sum_{i\in\operatorname{support}(\widehat{x})}d_{i}$.

Now by our choice of $\rho\geq\rho(\delta)$ (6) we have

[TABLE]

where the second step follows from (5). Furthermore, by Theorem 7, $\operatorname{support}(\widehat{x})$ contains the target cluster $K$, so the errors in $\operatorname{support}(\widehat{x})$ are solely attributed to the presence of false positives. Denoting by FP the set of false positives in $\operatorname{support}(\widehat{x})$ we can write

[TABLE]

Since $\textnormal{Vol}(K)=\sum_{i\in K}d_{i}\geq(1-\delta)k\bar{d}$ by Lemma 14, it follows that

[TABLE]

with probability at least $1-2\exp(-c_{0}\delta^{2}\bar{d})$. This proves the theorem.

B.4 Proof of Theorem 9

Under Assumption 1, with nonvanishing probability we can find a good seed node in the target cluster that is connected solely to $K$.

Lemma 21

Suppose that Assumption 1 holds, i.e., $q=\frac{c}{n}$ for a fixed constant $c>0$. Under the local random model given in Definition 6, if $n$ is sufficiently large, then

[TABLE]

We select this node as a seed node in the $\ell_{1}$-regularized PageRank problem.

Next we show that under the conditions of Theorem 9, the optimal solution to the reduced problem, $\widehat{x}^{\textnormal{R}}=\widehat{x}^{\textnormal{R}}(\rho(\delta))$ in (A.3), obeys

[TABLE]

Note that the above inequality is simply the optimality condition for the full-dimensional problem on $K^{c}$ (see Lemma 3). Since, by the optimality conditions for the reduced problem, $\widehat{x}^{\textnormal{R}}$ also satisfies the full-dimensional optimality condition on $K$. By uniqueness of the solution this means that $\widehat{x}^{\textnormal{R}}=\widehat{x}$. Also, since $\operatorname{support}(\widehat{x}^{\textnormal{R}})\subseteq K$ by construction, this proves that there is no false positive in $\widehat{x}$.

Now write $A_{j,K}\widehat{x}^{\textnormal{R}}$ into the sum of two terms,

[TABLE]

The first term can be ignored, since by Lemma 21 we choose the seed node to be the one that is solely connected to $K$. For the second term we use Hölder’s inequality to bound

[TABLE]

Applying Lemma 15, 19, and using the fact that $u\leq\frac{2\alpha}{(1+\alpha)\bar{d}},v\leq\frac{(1-\alpha)p}{(1+\alpha)\bar{d}^{2}}$ (which can be deduced from Lemma 16), and that $\delta\leq 0.1$ while $\alpha\in[0.1,0.9]$ (by assumptions of the theorem), then

[TABLE]

where $C_{1}>0$ is a positive constant. Then, to prove (16), it suffices to show

[TABLE]

Plugging in the definition of $\rho(\delta)$ (6) to the above, and using $\alpha\in[0.1,0.9]$ and $\delta\leq 0.1$, we obtain

[TABLE]

for some constant $C_{2}>0$. By the assumption (8) this means that $\rho(\delta)^{-1}\alpha^{-1}|(Q\widehat{x}^{\textnormal{R}}-\alpha\mathbf{s})_{j}|<d_{j}$, and thus we have proved the claim (16).

B.5 Proof of Theorem 11

First, by Theorem 5, we know that

[TABLE]

Since $\widehat{x}(\rho(\delta))$ contains the target cluster by Theorem 7, therefore together with the relation above we obtain $K\subseteq\operatorname{support}(x^{\text{APPR}})$. To see that the false positives in $x^{\text{APPR}}$ are also bounded, we can follow the same step as the proof of Theorem 8 (see Section B.3) to show that

[TABLE]

From the relation between $\operatorname{support}(x^{\text{APPR}}(\rho(\delta)))$ and $\operatorname{support}(\widehat{x}((1-\alpha)\cdot\rho(\delta)/2))$ it directly follows that

[TABLE]

Finally to prove the exact recovery case, it suffices to show that $\operatorname{support}(\widehat{x}((1-\alpha)\cdot\rho(\delta)/2))$ contains no false positives. In order for this all we need to do is to replace $\rho(\delta)$ by $\rho((1-\alpha)/2\cdot\delta)$ in the proof of Theorem 9. Then the same proof steps go through and we finally get the condition

[TABLE]

in place of (8). This completes the proof of the theorem.

C Proofs of lemmas

We prove all of our lemmas in this section.

C.1 Proof of Lemma 2

By the KKT condition $\widehat{x}$ must satisfy

[TABLE]

or equivalently,

[TABLE]

where $d=\operatorname{diag}(D)$ is the degree vector of the graph. Simple calculation shows that

[TABLE]

Now assume $j^{\star}\in V$ is a node such that $\widehat{x}_{j^{\star}}<0$ and that $j^{\star}=\operatorname*{arg\,min}_{j\in V}\widehat{x}_{j}$. Then from the condition (17), it must be that $(Q\widehat{x})_{j^{\star}}>0$. However by (C.1) this is only possible if there is at least one node $i\in V$ with $i\sim j^{\star}$ such that

[TABLE]

However this means that $\widehat{x}_{i}$ is smaller than $\widehat{x}_{j^{\star}}$, contradicting to the fact that $j^{\star}\in V$ is the smallest node in the graph. This proves the desired result.

C.2 Proof of Lemma 3

This lemma follows from Lemma 2 and the definition of $Q$.

C.3 Proof of Lemma 4

First, by Rosset and Zhu (2007, Proposition 1), we know that the optimal solution path $\widehat{x}(\rho)$ for the $\ell_{1}$-regularized minimization (2) is piecewise linear as a function of $\rho>0$. In particular, this implies that the path is continuous.

Next, we prove that if the support set of $\widehat{x}(\rho)$ remains constant in the interval $[\rho_{1},\rho_{0}]$ ($\rho_{0}>\rho_{1}$), then $\widehat{x}(\rho)$ is strictly decreasing on $[\rho_{1},\rho_{0}]$, i.e., $\widehat{x}(\rho_{1})>\widehat{x}(\rho_{0})$, where the inequality applies component-wise. This, together with the non-negativity of the solution (Lemma 2) and the continuity of the path $\widehat{x}(\rho)$, then establishes the lemma.

Write $\mathcal{A}(\rho)=\{j:\widehat{x}_{j}(\rho)>0\}$ for $\rho>0$ and choose $\rho_{0},\rho_{1}>0$ with $\rho_{0}>\rho_{1}$ such that the support set $\mathcal{A}(\rho)=\mathcal{A}$ does not change for $\rho\in[\rho_{1},\rho_{0}]$. For $j\in\mathcal{A}$, by Lemma 3, we have that

[TABLE]

and so

[TABLE]

Multiplying $Q_{\mathcal{A},\mathcal{A}}^{-1}$ on both sides (note that $Q_{\mathcal{A},\mathcal{A}}$ is positive definite),

[TABLE]

Now write

[TABLE]

For all $z\in\mathbb{R}^{\left\lvert{\mathcal{A}}\right\rvert}$, we have that

[TABLE]

where the second step holds since $d_{i}>\sum_{j:(i,j)\in E,j\in\mathcal{A}}w_{ij}$. This shows that all the eigenvalues of $(D^{-1/2}AD^{-1/2})_{\mathcal{A},\mathcal{A}}$ have absolute value less than $1$. Using the previous fact, $Q_{\mathcal{A},\mathcal{A}}^{-1}$ can be written as

[TABLE]

Now it is easy to see that the right hand side in (19) is positive, i.e., $\alpha(\rho_{0}-\rho_{1})Q_{\mathcal{A},\mathcal{A}}^{-1}d_{\mathcal{A}}>0$. This completes the proof of Lemma 4.

C.4 Proof of Lemma 13

This lemma is proved in Vershynin (2018, Proposition 2.4.1) which we reproduce here for completeness. By Chernoff’s inequality ((Vershynin, 2018, Theorem 2.3.1)), for some constant $c_{0}>0$,

[TABLE]

Using union bound,

[TABLE]

Since $\log k\leq c_{0}\delta^{2}\mu$ by assumption, the result follows.

C.5 Proof of Lemma 14

This lemma follows directly from Lemma 13.

C.6 Proof of Lemma 15

We have $\lVert{A_{j,K}}\rVert_{1}\sim\text{Binom}(k,q)$ for $j\in K^{c}$, so using tail bound for Binomial distribution ((Arratia and Gordon, 1989)), for any $t>c$,

[TABLE]

Set $t=c+3$. Then, since $\log(1-x)\geq-x$ for all $0<x<\frac{1}{2}$, then

[TABLE]

where the second inequality follows, since $k\geq 2(c+3)$ by assumption, and that $q\leq 0.5$ for $n$ sufficiently large. Then using union bound,

[TABLE]

as long as $n\gg k$.

C.7 Proof of Lemma 16

First the results in Lemma 2 and Lemma 3 hold for any graph, seed vector $\mathbf{s}$, and $\rho>0$, so the result can also be applied to (10). In particular, the solution $x^{\star}$ is non-negative, and thus the optimality condition can be expressed as

[TABLE]

(Note $\mathbf{s}$ is $1$ on the seed node and [math] on the rest.) For $\rho>\frac{1}{\bar{d}}$ the optimal solution (10) is the zero vector. For $\rho\leq\frac{1}{\bar{d}}$ the first nonzero nodes appear. From (21), as $\rho$ decreases less than or equal to $\frac{1}{\bar{d}}$, we can see that the seed node becomes active at first.

Now let $\rho^{\natural}>0$ be the regularization parameter for which the next set of nodes enters the active set. Then, for $\rho\in(\rho^{\natural},\frac{1}{\bar{d}}]$, we know that $x^{\star}_{S}>0$ and $x^{\star}_{S^{c}}=0$. Writing $x^{\star}_{S}=u\mathbf{1}_{S}$ for some $u>0$, we can see that for the seed node $j\in S$,

[TABLE]

Substituting in the optimality condition (21), and solving with respect to $u$, we get

[TABLE]

It remains to be shown that $x^{\star}_{S}=u\mathbf{1}_{S}$, for the above $u$, satisfies the optimality conditions for $j\in S^{c}$. To verify this, we use the definition of $u$ and the optimality conditions for $j\in S^{c}$. We need

[TABLE]

We have that

[TABLE]

and

[TABLE]

Using the above equalities, we get that the optimality conditions are satisfied if and only if

[TABLE]

Next, we prove that, when $\rho$ reaches $\rho^{\natural}$, the nodes in $K\backslash S$ appear in the active set. To see this, from (21), we can check that the first $\rho$ value allowing nonzero nodes for $K\backslash S$ is given by

[TABLE]

Now assuming that $x^{\star}$ takes the form of $u\mathbf{1}_{S}+v\mathbf{1}_{K}$, and substituting in the optimality condition (21), we obtain the following equations:

[TABLE]

So, solving with respect to $u$ and $v$, we obtain

[TABLE]

Also, $x^{\star}=u\mathbf{1}_{S}+v\mathbf{1}_{K}$, with $u$ and $v$ given in (23), satisfies the optimality conditions for $K^{c}$ if and only if

[TABLE]

where $\mathbb{E}\left[{{d}_{\ell_{\sharp}}}\right]=\min_{\ell\in K^{c}}\mathbb{E}\left[{d_{\ell}}\right]$. We claim that $\rho^{\natural}>\rho^{\sharp}$, in which case we have proved that $x^{\star}=u\mathbf{1}_{S}+v\mathbf{1}_{K}$ satisfies the optimality condition (21) for $\rho\in[\rho^{\sharp},\rho^{\natural})$, which proves the theorem.

It now remains to check that $\rho^{\natural}>\rho^{\sharp}$. With some algebra, we have that

[TABLE]

Also, since $\bar{d}=p(k-1)+q(n-k)$, we have

[TABLE]

and so

[TABLE]

Under Assumption 1, we know that $p\mathbb{E}\left[{d_{\ell_{\sharp}}}\right]=\mathcal{O}\left({1}\right)$ while $q\bar{d}=\mathcal{O}\left({\frac{\bar{d}}{n}}\right)$, so $p\mathbb{E}\left[{{d}_{\ell_{\sharp}}}\right]-q\bar{d}>0$ for a sufficiently large $n$. This verifies (24).

C.8 Proof of Lemma 17

Define

[TABLE]

We will prove that $\check{x}^{\textnormal{R}}=\check{x}^{\textnormal{R}}(\rho)>0$ for any $\rho\leq\rho(\delta)$ where $\rho(\delta)$ is defined in (6). In particular, this means that $\check{x}^{\textnormal{R}}$ satisfies the optimality condition for the reduced problem (A.3), implying that $\widehat{x}^{\textnormal{R}}_{K}=\check{x}^{\textnormal{R}}>0$ and thus proving the result.

First, using (C.3), we can write

[TABLE]

Let $w=D_{K,K}^{-1}\mathbf{s}_{K}$. Then

[TABLE]

Note that $\mathbf{s}_{K}$ has mass $1$ on the seed node, so we have $w=\mathbf{s}_{K}/d_{S}$, where $d_{S}$ is the degree of the seed node. Denoting by FON the first-order neighbors of the seed node in $K$, i.e., \text{FON}=\{i\in K:\text{i is a neighbor of the seed node}\}, we then have

[TABLE]

where $\mathbf{1}_{\text{FON}}$ is the indicator vector for the set FON. Applying the degree concentration Lemma 14, then with probability at least $1-2e^{-c_{0}\delta^{2}\bar{d}}$,

[TABLE]

Next, using Lemma 13, we have that $|\text{FON}|\geq(1-\delta)pk$ with probability at least $1-2e^{-c_{0}\delta^{2}pk}$. Furthermore, for each non-seed node $i\notin\text{FON}$ in $K$, it is connected to nodes in FON with probability $p$, independently of all other pairs. Then $A_{i,K}^{\top}\mathbf{1}_{\text{FON}}|\;|\text{FON}|\stackrel{{\scriptstyle\mathrm{iid}}}{{\sim}}\text{Binom}(|\text{FON}|,p)$, and thus applying Lemma 13, and using the assumption $(1-\delta)p^{2}k\geq c_{0}^{-1}\delta^{-2}\log k$, we get

[TABLE]

Hence, we can conclude that with probability at least $1-4e^{-c_{0}\delta^{2}(1-\delta)p^{2}k}$,

[TABLE]

Then,

[TABLE]

where the first step applies (26), the second step uses (27), and the third applies Lemma 14 and the fact that $p\cdot k=\gamma\bar{d}$. Returning to (C.8), we can now see that $\check{x}^{\textnormal{R}}>0$ as long as $\rho$ is less than $\rho(\delta)=\left(\frac{1-\alpha}{1+\alpha}\right)^{2}\left(\frac{1-\delta}{1+\delta}\right)^{2}\frac{\gamma p}{(1+\delta)\bar{d}^{2}}$. This completes the proof of Lemma 17.

C.9 Proof of Lemma 18

Recall that $\widehat{x}$ is the $\ell_{1}$-regularized PageRank on the original graph (2). If $\rho>1/d_{S}$, then we can easily check that $\widehat{x}=\widehat{x}^{\textnormal{R}}=0$, while for $\rho=1/d_{S}$, we have $\operatorname{support}(\widehat{x}^{\textnormal{R}})=\operatorname{support}(\widehat{x})=S$. To prove that $\operatorname{support}(\widehat{x}^{\textnormal{R}})\subseteq\operatorname{support}(\widehat{x})$ holds for $\rho<1/d_{S}$, we proceed by induction.

Writing $\mathcal{A}_{1}=\operatorname{support}(\widehat{x}^{\textnormal{R}})$ and $\mathcal{A}_{2}=\operatorname{support}(\widehat{x})$, by the optimality condition, we have

[TABLE]

By inductive hypothesis, assume $\mathcal{A}_{1}\subseteq\mathcal{A}_{2}$. According to the expression (C.3), then we can see that $\widehat{x}_{\mathcal{A}_{1}}\geq\widehat{x}^{\textnormal{R}}_{\mathcal{A}_{1}}$, where the inequality applies component-wise. Now let $i\in K$ be a node such that $\widehat{x}^{\textnormal{R}}_{i}=\widehat{x}_{i}=0$. Using the optimality condition, we know that $i$ becomes active for the full-dimensional problem (2) whenever the following condition is satisfied:

[TABLE]

Analogously, the node $i$ becomes active for the reduced problem (A.3) when the following condition is satisfied:

[TABLE]

Comparing the left-hand sides in (28) and (29), it is obvious that under the induction assumption, the left-hand side in (28) is larger than in (29). This implies that $\widehat{x}_{i}$ becomes active earlier than $\widehat{x}^{\textnormal{R}}_{i}$, so the induction assumption continues to hold. This completes the proof of the lemma.

C.10 Proof of Lemma 19

Since $\widehat{x}^{\textnormal{R}}=(\widehat{x}^{\textnormal{R}}_{S},\widehat{x}^{\textnormal{R}}_{K\backslash S})$ is the solution to the reduced problem (A.3), fixing $\widehat{x}^{\textnormal{R}}_{S}$, then $\widehat{x}^{\textnormal{R}}_{K\backslash S}$ is the minimizer of the following optimization problem:

[TABLE]

Due to Lemma 17, the subgradient of $\lVert{D\cdot(\widehat{x}^{\textnormal{R}}_{S},y)}\rVert_{1}$ at $y=\widehat{x}^{\textnormal{R}}_{K\backslash S}$ is given by $d_{K\backslash S}$. Using optimality, it follows that

[TABLE]

Next, we can prove that our choice of $\rho=\rho(\delta)$ lies in the interval $[\rho^{\sharp},\rho^{\natural})$.

Lemma 22

Under the conditions of Lemma 19, we have $\rho=\rho(\delta)\in(\rho^{\sharp},\rho^{\natural})$.

Hence, by Lemma 16, the population version of $\ell_{1}$-regularized PageRank (10) has support $K$, so it follows that $x^{\star}$ is also the solution to the reduced problem

[TABLE]

(Abusing notation, we use $x^{\star}$ and $x^{\star}_{K}$ interchangeably throughout the proof.) So, using the optimality condition, we obtain

[TABLE]

where we have $\partial\lVert{\mathbb{E}\left[{D}\right]\cdot(x^{\star}_{S},y)}\rVert_{1}=\mathbb{E}\left[{d_{K\backslash S}}\right]$ due to Lemma 16 and Lemma 22. Combining the two optimality equations, and adding and subtracting $Q_{K\backslash S,K}x^{\star}$, it follows that

[TABLE]

Writing $Q_{K\backslash S,K}(x^{\star}-\widehat{x}^{\textnormal{R}})=Q_{K\backslash S,S}(x^{\star}_{S}-\widehat{x}^{\textnormal{R}}_{S})+Q_{K\backslash S,K\backslash S}(x^{\star}_{K\backslash S}-\widehat{x}^{\textnormal{R}}_{K\backslash S})$, and rearranging terms, we get

[TABLE]

Taking the $\ell_{\infty}$ norm on both sides,

[TABLE]

For term 1 we know that $x^{\star}=u\mathbf{1}_{S}+v\mathbf{1}_{K}$ from Lemma 16, so plugging in to term 1,

[TABLE]

Thus,

[TABLE]

with probability at least $1-4e^{-c_{0}\delta^{2}pk}$, where the second step uses the triangle inequality, the third step applies Lemma 13 and Lemma 14, and the last step holds since $\lvert{A_{j,S}-p}\rvert\leq 1$ for all $j\in K\backslash S$. Furthermore, by Lemma 14, we can bound term 2 as

[TABLE]

while for term 3, simple calculation leads to

[TABLE]

Putting the results together, we have

[TABLE]

Now it remains to upper bound the term $\left\lvert{\widehat{x}^{\textnormal{R}}_{S}-x^{\star}_{S}}\right\rvert$. Following the same steps as before, indeed we can check that the following holds:

[TABLE]

Also, we can see that using $x^{\star}=u\mathbf{1}_{S}+v\mathbf{1}_{K}$,

[TABLE]

where the inequality applies Lemma 13 and Lemma 14. It is also easy to see that $\text{term 5}\leq\rho\alpha\cdot\delta\bar{d}$, while

[TABLE]

where the second step uses Hölder’s inequality and the next step applies Lemma 13. Furthermore, $Q_{S,S}=\frac{1+\alpha}{2}d_{S}$ by definition, which is bounded below by $\frac{1+\alpha}{2}(1-\delta)\bar{d}$. Thus, from (34), we get

[TABLE]

Finally, return to (33) and substitute the above bound in place of $\lVert{\widehat{x}^{\textnormal{R}}_{S}-x^{\star}_{S}}\rVert_{\infty}$, then

[TABLE]

The following lemma concerns the local strong convexity of the matrix $Q_{K\backslash S,K\backslash S}$ in $\ell_{\infty}$ norm.

Lemma 23

Under the conditions of Lemma 19, with probability at least $1-2e^{-c_{0}\delta^{2}\bar{d}}$,

[TABLE]

Applying Lemma 23 to the left-hand side of (35), and rearranging terms and simplifying, we have that

[TABLE]

where we use $\alpha<1$ and $p(k-1)=\gamma\bar{d}$. By assumption of the lemma, we have that $\delta\leq 0.1$ and that $\alpha\in[0.1,0.9]$ while $\gamma\in(0,1)$ by definition (5), so

[TABLE]

for some numerical constant $c_{1}>0$. As a result,

[TABLE]

Dividing both sides by $c_{1}\bar{d}$,

[TABLE]

Using the triangle inequality, Lemma 19 now follows.

C.11 Proof of Lemma 20

By Lemma 3 we know that

[TABLE]

Summing over $i$’s, we have

[TABLE]

where the first equality holds since $\mathbf{1}^{\top}Q\widehat{x}=\mathbf{1}^{\top}(\alpha D+(1-\alpha)/2\cdot L)\widehat{x}=\alpha\sum_{i}d_{i}\widehat{x}_{i}$. Dividing both sides by $-\rho\alpha$, we get

[TABLE]

C.12 Proof of Lemma 21

This result follows from simple probability calculation. We have

[TABLE]

Here, the second and fourth steps hold since each pair of nodes has an edge independently of all other pairs. Now suppose that $n$ is sufficiently large so that $q\leq 0.5$. Then, we have $(1-q)^{n}\geq\exp(-1.5c)$, which leads to

[TABLE]

C.13 Proof of Lemma 22

Letting $\bar{c}=\left(\frac{1-\alpha}{1+\alpha}\right)\left(\frac{1-\delta}{1+\delta}\right)^{2}\frac{\gamma}{1+\delta}$, we can write $\rho=\rho(\delta)=\frac{\bar{c}(1-\alpha)p}{(1+\alpha)\bar{d}^{2}}$. Following the same steps as (24) (proof of Lemma 16), we can see

[TABLE]

Since $\delta\leq 0.1$ and $\alpha\in[0.1,0.9]$, we know that $\bar{c}\geq c^{\prime}\gamma$ for some constant $c^{\prime}>0$. Also, since $q=\frac{c}{n}$ while $p=k=\mathcal{O}\left({1}\right)$ by Assumption 1, we have that $\gamma=\mathcal{O}\left({1}\right)$, and thus

[TABLE]

Therefore, if $n$ is sufficiently large, the first two terms on the right-hand side of (C.13) are positive and much larger than the last term. This proves that $\rho>\rho^{\sharp}$.

To see $\rho<\rho^{\natural}$, it suffices to check

[TABLE]

Rearranging the terms, we equivalently need to show $2\alpha(1+\alpha)\bar{d}>(1-\alpha)^{2}p$. Since $\bar{d}\geq p\cdot(k-1)$, we then have $2\alpha(1+\alpha)\cdot(k-1)>(1-\alpha)^{2}$ which, due to the assumption $\alpha\in[0.1,0.9]$, holds for any $k\geq 5$.

C.14 Proof of Lemma 23

Denoting by $\mathbf{1}$ the vector whose entries are all ones, we have

[TABLE]

where $L_{K\backslash S,K\backslash S}$ is the graph Laplacian of the sub-graph induced by $A_{K\backslash S,K\backslash S}$.

Now assume $\lVert{y}\rVert_{\infty}=1$. Without loss of generality, we will assume $y_{1}=1$ (the same argument holds in the case of $y_{1}=-1$). Then, by definition of graph Laplacian, it is easy to see $(L_{K\backslash S,K\backslash S}\cdot y)_{1}\geq 0$, and so

[TABLE]

Hence, applying degree concentration to $d_{1}$ (Lemma 14),

[TABLE]

with probability at least $1-2e^{-c_{0}\delta^{2}\bar{d}}$, proving the result.