Pairwise Link Prediction

TL;DR

This paper introduces a new pairwise link prediction task focused on predicting the formation of triangles in networks, and develops PageRank-based methods that outperform traditional approaches in various network types.

Contribution

The paper proposes a novel pairwise link prediction framework targeting triangle formation and extends existing methods with PageRank-based algorithms.

Findings

Diffusion-based methods are more consistent across different network types.

The pairwise link prediction framework improves standard link prediction accuracy.

PageRank-based methods outperform traditional approaches in experiments.

Abstract

Link prediction is a common problem in network science that transects many disciplines. The goal is to forecast the appearance of new links or to find links missing in the network. Typical methods for link prediction use the topology of the network to predict the most likely future or missing connections between a pair of nodes. However, network evolution is often mediated by higher-order structures involving more than pairs of nodes; for example, cliques on three nodes (also called triangles) are key to the structure of social networks, but the standard link prediction framework does not directly predict these structures. To address this gap, we propose a new link prediction task called "pairwise link prediction" that directly targets the prediction of new triangles, where one is tasked with finding which nodes are most likely to form a triangle with a given edge. We develop two PageRank-based methods for our pairwise link prediction problem and make natural extensions to existing link prediction methods. Our experiments on a variety of networks show that diffusion based methods are less sensitive to the type of graphs used and more consistent in their results. We also show how our pairwise link prediction framework can be used to get better predictions within the context of standard link prediction evaluation.

Tables

Table 1. Table 1 : Statistics of the real-world datasets used in this paper.

Network name	nodes	edges	triangles	type
Penn94	41536	1362220	7207796	Social
Caltech36	762	16651	119562	Social
Ch-Ch-Miner	1510	48512	568466	Biology
P-P-Pathways	21521	338624	2394642	Biology
email	1133	5451	5343	Communication
CollegeMsg	1899	13838	14319	Temporal
Email-EU	1005	32128	105461	Temporal

Table 2. Table 2 : description of methods inspired by pairwise link to perform the standard link prediction task

sum$▲$	For a certain node $i$, aggregate the pair-seeded PageRank results from all edges adjacent to $i$. This is equivalent to performing PageRank with a normalized initial vector valued $1$ at the indices of all the neighbors of $i$, and $\text{degree}\left(i\right)$ at index $i$.
max$\bullet$	This is similar to the previous approach, but here, we instead take the element-wise maximum value of the pair-seeded PageRank vectors.
star-seed+	This is similar to pair-seeded PageRank, except that we start PageRank with a normalized initial vector valued $1$ at the index of the seed node and all its neighbors.
TRPR$◆$	This uses the same starting vector used by star-seed, but instead, applies the TRPR algorithm on it.

Equations

$$\frac{|\Gamma (w)\cap \Gamma (u)|}{|\Gamma (w)\cup \Gamma (u)|}.$$

$$\sum _{z\in \Gamma (w)\cap \Gamma (u)}\frac{1}{\log (|\Gamma (z)|)}.$$

$$|\Gamma (w)|\cdot |\Gamma (u)|.$$

$$\Gamma ((u,v))$$

$$=\Gamma (u)\cup \Gamma (v)\setminus \{u,v\}.$$

$$\text{JS}(w,(u,v))$$

$$\text{AA}(w,(u,v))$$

$$\text{PA}(w,(u,v))$$

$$\text{JS–MAX}(w,(u,v))$$

$$\text{JS–MUL}(w,(u,v))$$

$$\text{AA–MAX}(w,(u,v))$$

$$\text{AA–MUL}(w,(u,v))$$

$$(\mathbf{I}-\alpha \mathbf{P})\mathrm{x}=(1-\alpha ){\mathrm{e}}_{u,v}.$$

$$(\mathbf{I}-\alpha \mathbf{P}){\mathrm{x}}_u$$

$$(\mathbf{I}-\alpha \mathbf{P}){\mathrm{x}}_v$$

$$(\mathbf{I}-\alpha \mathbf{P})({\mathrm{x}}_u+{\mathrm{x}}_v)$$

$$\frac{1}{2}(\mathbf{I}-\alpha \mathbf{P})({\mathrm{x}}_u+{\mathrm{x}}_v)$$

$$(\mathbf{I}-\alpha \mathbf{P})\mathrm{x}$$

$$\mathchoice{\hbox to0.0pt{{$\underline{{\hbox to4.6289pt{}}}$\hss}}}{\hbox to0.0pt{{$\underline{{\hbox to4.6289pt{}}}$\hss}}}{\hbox to0.0pt{{$\underline{{\hbox to3.24023pt{}}}$\hss}}}{\hbox to0.0pt{{$\underline{{\hbox to2.31444pt{}}}$\hss}}}\bm{{T}}(i,j,k)=\begin{cases}1&\text{if $(i,j,k)$ is a triangle}\\ 0&\text{otherwise}.\end{cases}$$

$${\mathrm{x}}_i$$

$$\bm{\mathrm{x}}_{i}=\alpha\mathchoice{\hbox to0.0pt{{$\underline{{\hbox to4.6289pt{}}}$\hss}}}{\hbox to0.0pt{{$\underline{{\hbox to4.6289pt{}}}$\hss}}}{\hbox to0.0pt{{$\underline{{\hbox to3.24023pt{}}}$\hss}}}{\hbox to0.0pt{{$\underline{{\hbox to2.31444pt{}}}$\hss}}}\bm{{T}}[\bm{\mathrm{x}}_{i-1}]\bm{{D}}_{A_{i}}^{-1}\bm{\mathrm{x}}_{i-1}+\alpha\bm{{A}}\bm{{D}}_{A_{i}}^{-1}\bm{\mathrm{x}}_{i-1}+(1-\alpha)\bm{\mathrm{x}}_{0}.$$

$$\bm{\mathrm{x}}_{i}=\alpha\mathchoice{\hbox to0.0pt{{$\underline{{\hbox to4.6289pt{}}}$\hss}}}{\hbox to0.0pt{{$\underline{{\hbox to4.6289pt{}}}$\hss}}}{\hbox to0.0pt{{$\underline{{\hbox to3.24023pt{}}}$\hss}}}{\hbox to0.0pt{{$\underline{{\hbox to2.31444pt{}}}$\hss}}}\bm{{T}}[\bm{\mathrm{x}}_{i-1}]\bm{\mathrm{y}}_{i-1}+\alpha\bm{{A}}\bm{\mathrm{y}}_{i-1}+(1-\alpha)\bm{\mathrm{x}}_{0}.$$

$$\text{MAX}(u,v)$$

$$\text{MUL}(u,v)$$

$$\text{SP}((u,v),k)=\{\begin{array}{ll}1& \text{if at least one ground truth node w appears in the top kmissing}\\ & \text{predictions for edge (u,v)missing}\\ 0& \text{otherwise}.\end{array}$$

Full text

1   Introduction

Networks are a standard tool for data analysis in which links between data points are the primary object of study. A fundamental problem in network analysis is link prediction Liben-Nowell and Kleinberg [2007]; Lü and Zhou [2011], which is typically formulated as a problem of identifying pairs of nodes that will either form a link in the future (when viewing the network as evolving over time) or whose connection is missing from the data Clauset et al. [2008]. The link prediction problem has applications in a variety of domains. For instance, in online social networks of friendships, predicting that two people will form a connection can be used for friendship recommendation Backstrom and Leskovec [2011]. Similarly, predicting new links between users and items on platforms such as Amazon and Netflix can be used for product recommendation Gomez-Uribe and Hunt [2015]. And in biology, link prediction is used to identify novel interactions between genes, diseases, and drugs within interaction networks Lin et al. [2018]. In the settings above, the link prediction problem is oriented around—and evaluated in terms of—the identification of pairs of nodes that are likely to be connected. However, there is mounting evidence that the organization and evolution of networks is centered around higher-order interactions involving more than two nodes Milo et al. [2002]; Milo [2004]; Benson et al. [2016]; Benson et al. [2018]; Lambiotte et al. [2019]. In the case of social networks, triangles (cliques on three nodes) are extremely common due to various sociological mechanisms driving triadic closure Easley et al. [2010]; Holland and Leinhardt [1977]; Granovetter [1977]; Rapoport [1953]. Methods for link prediction are indeed motivated by these ideas. For instance, the Jaccard similarity between the sets of neighbors of two nodes—a common heuristic for link prediction Liben-Nowell and Kleinberg [2007]—measures the number of triangles that would be created if the two nodes are linked, normalized by the total number of neighbors of the two nodes. Still, such methods are used to make predictions on pairs of nodes, rather than a prediction on the appearance of the higher-order structures directly. Here, we develop a framework for directly predicting the appearance of a higher-order structure. We focus on the case of triangles, which is one of the simplest higher-order structures while also being critical to social network analysis. Again, classical link prediction is centered around the following question: given a node $u$ in the network, which nodes are likely to link to $u$? This scenario is illustrated in Figure 1. Our framing of the problem is similar, but we instead ask the following: given an edge $(u,v)$ in the network, which nodes are likely to connect to both $u$ and $v$? We call this the pairwise link prediction problem, and it is illustrated in Figure 1. There are several scenarios where the pairwise link prediction problem is natural, such as recommending a new friend to a couple on an online social network, recommending a movie to a couple in a video site, or predicting an effective drug given a disease-gene pair. We devise two new algorithms for the pairwise link prediction problem. The first is based on a variant of seeded (personalized) PageRank that uses multiple seeds, namely, one seed at each end point of the edge for which we are trying to predict new triadic connections. The second is based on a PageRank-like iteration that puts more weight on edges that participate in many triangles. In this sense, the method reinforces triangles, and we call the method “Triangle Reinforced PageRank” (TRPR). We compare these algorithms to natural extensions of local similarity measures that are common in link prediction, such as Jaccard similarity Liben-Nowell and Kleinberg [2007], Adamic-Adar similarity Adamic and Adar [2003], and preferential attachment Newman [2001]. For a given edge, each of the above methods produces a score for the remaining nodes in the graph. We find that our proposed diffusion based methods are the least sensetive to the graph type and degree distribution and often produce the top results. We provide code for all the methods used in this paper in the repository:

[TABLE]

††margin:

Figure 1: (A) In standard link prediction, we are tasked with finding nodes that are likely to link to a given node $u$. (B) In this paper, we study pairwise link prediction, where we are tasked with finding nodes that are likely to form a triangle with a given edge $(v,w)$.

2   Background and Related Work

We now briefly review some related work in link prediction and higher-order structure. As part of this, we will go over methods that we will generalize in the next section for the pairwise link prediction problem. All of these methods assign some similarity score between pairs of nodes, where a larger similarity is indicative of pairs that are likely to connect. For notation, we use $\Gamma(u)$ to denote the set of neighbors of node $u$ in the graph.

2.1  Local methods

Several approaches to link prediction are based on local information in the graph, namely a score is assigned to a pair of nodes $w$ and $u$ based on their 1-hop neighborhoods $\Gamma(w)$ and $\Gamma(u)$. One approach that falls under this category stems from the idea that as $\lvert\Gamma(w)\cap\Gamma(u)\rvert$ increases, the chance that $u$ and $v$ are connected also increases Newman [2001]. Here, $\lvert\Gamma(w)\cap\Gamma(u)\rvert$ is the number of triangles that would be formed if $u$ and $v$ were connected. Often, this number is normalized by the size of the neighborhoods, which gives rise to the Jaccard similarity between two nodes $w$ and $u$:

[TABLE]

The Adamic–Adar similarity measure Adamic and Adar [2003] is a local score that assigns similarity between two nodes based on how important their common neighbors are, where importance is measured by the degree of a given node. Formally, the Adamic–Adar similarity measure between nodes $u$ and $v$ is:

[TABLE]

A third local method is based on preferential attachment, where nodes are more likely to connect to established nodes in the network, and, established nodes have a higher chance to connect to each other Barabási and Albert [1999]; Newman [2001]. Using degree as a proxy for how established a node is, the preferential attachment score between nodes $w$ and $u$ is:

[TABLE]

2.2  Global methods

Another set of approaches for link prediction are based on aggregating (weighted or normalized) path counts of varying lengths. In contrast to the local methods described above, these methods use global information about the entire network. For example, the Katz similarity counts the number of paths between two nodes, weighting paths of length-$k$ by $\beta^{k}$ Katz [1953]; Liben-Nowell and Kleinberg [2007]. Another class of global methods are methods based on conservative diffusions such as PageRank Page et al. [1999]. Such diffusion methods are typically seeded by a particular node $u$, and the similarity of $u$ to all other nodes is given by the amount of “mass” that diffuses to each other node. We will make use of PageRank-like methods in the next section.

2.3  Higher-order structure

Since a network encodes pairwise relationships (edges) between elements (nodes), the link prediction problem is natural in many cases. Nevertheless, recent studies have shown that networks evolve through higher-order interactions, i.e., much of the structure in evolving networks involves interactions between more than just two nodes Benson et al. [2018]. Recent research has also introduced the problem of predicting the time when an edge addition will close a triangle Dave and Hasan [2019]. Furthermore, random graph models constructed from distributions of triangles have shown to be good fits for real-world data Eikmeier et al. [2018], providing additional evidence that triadic relationships are important to the assembly of networks.

3   Methods

We propose several methods for the pairwise link prediction problem. First, we extend the three local methods described above to measure node-edge similarity. After, we propose diffusion-based methods akin to seeded PageRank.

3.1  Local similarity measures for pairwise prediction

Our goal here is to extend common local methods for link prediction to the scenario of pairwise link prediction. In other words, instead of computing similarity between nodes, we now compute similarity between an edge and a node. To do this, we simply replace the neighborhood of one node with the neighborhood of an edge. This requires that we specify what the neighborhood of an edge $(u,v)$ should capture. We define:

[TABLE]

Note that this is akin to the boundary of a set of vertices in the graph that is often used to define the size of a cut, which — for an edge — would correspond to the union of neighborhoods. This definition is contrary to what we use in Nassar et al. [2019], where we define the neighborhood to be the intersection. Initially, the intersection of neighborhoods was a natural choice, but in practice the intersection set is very limiting specially in scenarios when an edge is connected to the rest of the graph, yet does not participate in any triangles. We should still be able to make predictions on such edges, and our choice of the union of neighborhoods handles this.

Using the substitution gives us three similarity measures that will compute the similarity of an edge to a node.

• ·

Jaccard Similarity (JS).

[TABLE]

• ·

Adamic–Adar (AA).

[TABLE]

• ·

Preferential Attachment (PA).

[TABLE]

Further, we extend the Jaccard Similarity and Adamic–Adar measures to account for a combination of the single link prediction results. We use the maximum value of the single similarity score of both end points of an edge $(u,v)$ with another node $w$, as well as the product of similarity values. We state these measure below.

• ·

Jaccard Similarity.

[TABLE]

• ·

Adamic–Adar.

[TABLE]

Next, we develop two new methods for pairwise link prediction based on seeded PageRank, and use a combination of the single seeded PageRank results to compute a new measure of similarity between an edge and a node.

3.2  Pair-seeded PageRank

Seeded PageRank is a foundational concept in network analysis that models a flow of information in a network to predict links and communities on a network Andersen et al. [2006]; Gleich [2015]. Seeded PageRank models information flow from the seed node to other nodes in the network via a Markov chain, and the stationary distribution of the chain provides the scores on the nodes. A high score on a node is a signal that the node should be connected to the seed node. More formally, let $\bm{{A}}$ be the symmetric adjacency matrix of an undirected graph, and let $\bm{{P}}$ be the column stochastic matrix of a random walk on that graph. Specifically, $P(i,j)=A(i,j)/\lvert\Gamma(j)\rvert$. Let $u$ be the seed node. Then the seeded PageRank scores are entries of the solution vector $\bm{\mathrm{x}}$ to the linear system $(\bm{{I}}-\alpha\bm{{P}})\bm{\mathrm{x}}=(1-\alpha)\bm{\mathrm{e}}_{u}$.

Here, $\bm{\mathrm{e}}_{u}$ is the vector of all zeros, except at index $u$, where $\bm{\mathrm{e}}_{u}(u)=1$ (i.e., $\bm{\mathrm{e}}_{u}$ is the indicator vector on node $u$). The parameter $\alpha$ is the probability of transitioning according to the probability distribution in $\bm{{P}}$ and $(1-\alpha)$ is the probability of teleporting according to the probability distribution in $\bm{\mathrm{e}}_{u}$. The entries of $\bm{\mathrm{x}}$ provide similarities between node $u$ and the other nodes and thus can be used for standard link prediction.

In the same way seeded PageRank predicts the relevance of other nodes in the network to a single seed node, we propose pair-seeded PageRank to predict the relevance of nodes to a single edge; with these similarities, we are able to make predictions for the pairwise link prediction problem. For a given edge $(u,v)$, pair-seeded PageRank solves the following linear system:

[TABLE]

In this case, $\bm{\mathrm{e}}_{u,v}$ is the vector of all zeros, except at indices $u$ and $v$, where $\bm{\mathrm{e}}_{u,v}(u)=\bm{\mathrm{e}}_{u,v}(v)=1/2$. The solution $\bm{\mathrm{x}}$ can be interpreted as the similarity of each node to the edge $(u,v)$.

We now note that pair-seeded PageRank is equivalent to the sum of single-seeded PageRank on each of the nodes, up to a scalar multiple. This follows quickly from linearity of the PageRank problem. To see this, let $\bm{\mathrm{x}}_{u}$ and $\bm{\mathrm{x}}_{v}$ be the seeded PageRank solutions corresponding to nodes $u$ and $v$ respectively. Then,

[TABLE]

Hence, $2\bm{\mathrm{x}}=\bm{\mathrm{x}}_{u}+\bm{\mathrm{x}}_{v}$, and the pair-seeded PageRank solution is equivalent to the summation of the single seeded PageRank equations, up to scaling. Indeed, this is a useful and helpful observation as there are many systems designed to estimate large seeded PageRank values for single-seeds by using highly scalable random walk methods Lofgren et al. [2016]. Thus, this technique could be used wherever a PageRank-style prediction is already employed.

3.3  Triangle Reinforced PageRank (TRPR)

We now propose a PageRank-like method that uses a weighting scheme on edges based on the number of triangles that contains each edge, which we call Triangle Reinforced PageRank (TRPR). For an unweighted graph, the PageRank solution is highly affected by the degree of nodes in the network. Here, we reinforce the influence of triangles by giving edges participating in many triangles a higher weight. Figure 3.3 presents a motivating example for the usefulness of reinforcing triangles. ††margin:

Figure 2: Motivating social network example for the TRPR algorithm. If all of the friends of the blue couple know the red node, we want to predict that the red node must know the blue couple as well. Running TRPR on the above example with $\bm{\mathrm{e}}_{u,v}$ as the seed vector on the blue nodes reveals that the red node has the third highest score after the two blue nodes. After $10$ iterations of Algorithm 1 with $\alpha=0.85$, the output vector assigns a score of $0.120$ to the red node, $0.062$ to the black nodes, and $0.252$ to the blue nodes.

To develop our TRPR method, we first introduce a tensor \mathchoice{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to3.24023pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to2.31444pt{}}}\hss}}}\bm{{T}}, that encodes all triangles in a network:

[TABLE]

Again, in our derivation, we assume that the graph is undirected so that \mathchoice{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to3.24023pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to2.31444pt{}}}\hss}}}\bm{{T}} is fully symmetric in all permutations of indices. A typical way to solve the PageRank linear system is the power method. With TRPR, we modify the power method by adding a step that redistributes the weights in the network. Specifically, we compute the matrix \hat{\bm{{X}}}=\mathchoice{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to3.24023pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to2.31444pt{}}}\hss}}}\bm{{T}}[\bm{\mathrm{x}}], where \hat{\bm{{X}}}(i,j)=\sum_{k}\mathchoice{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to3.24023pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to2.31444pt{}}}\hss}}}\bm{{T}}(i,j,k)\bm{\mathrm{x}}(k), which measures the relevance of edge $(i,j)$ to the distribution of node scores in the vector $\bm{\mathrm{x}}$. We then run an iteration of the power method on a weighted adjacency matrix $\bm{{X}}=\hat{\bm{{X}}}+\bm{{A}}$, where the columns are re-normalized to make the matrix column stochastic. Algorithm 1 shows the idealized algorithm.

TRPR can be implemented efficiently.  Although TRPR involves the tensor \mathchoice{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to3.24023pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to2.31444pt{}}}\hss}}}\bm{{T}}, we do not need to form it explicitly, and we show an alternative derivation here. We first unwrap one iteration of TRPR. Let \bm{{A}}_{i}=\mathchoice{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to3.24023pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to2.31444pt{}}}\hss}}}\bm{{T}}[\bm{\mathrm{x}}_{i-1}]+\bm{{A}}, then, at iteration $i$, we can translate $\bm{\mathrm{x}}_{i}=\alpha\bm{{P}}_{i}\bm{\mathrm{x}}_{i-1}+(1-\alpha)\bm{\mathrm{x}}_{0}$ into

[TABLE]

where $\bm{{D}}_{A_{i}}^{-1}$ is a diagonal matrix with the $i^{th}$ diagonal entry being the inverse of the sum of edge weights connected to node $i$ in $\bm{{A}}_{i}$ (again, we assume a connected graph so these values are all non-zero). Then,

[TABLE]

Set $\bm{\mathrm{y}}_{i-1}=\bm{{D}}_{A_{i}}^{-1}\bm{\mathrm{x}}_{i-1}$. Then

[TABLE]

The relevant computationally expensive pieces to compute are \mathchoice{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to3.24023pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to2.31444pt{}}}\hss}}}\bm{{T}}[\bm{\mathrm{x}}_{i-1}]\bm{\mathrm{y}}_{i-1} and the entries of $\bm{{D}}^{-1}_{A_{i}}$. Both involve the same type of operation. Using the definition of \mathchoice{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to3.24023pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to2.31444pt{}}}\hss}}}\bm{{T}}[\bm{\mathrm{x}}] we have that the matrix-vector product \bm{\mathrm{z}}=\mathchoice{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to3.24023pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to2.31444pt{}}}\hss}}}\bm{{T}}[\bm{\mathrm{x}}]\bm{\mathrm{y}} has z_{i}=\sum_{j}\sum_{k}\mathchoice{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to3.24023pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to2.31444pt{}}}\hss}}}\bm{{T}}(i,j,k)y(j)x(k). Consequently, if we have any means of iterating over the triangles of a graph, then we can compute \mathchoice{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to3.24023pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to2.31444pt{}}}\hss}}}\bm{{T}}[\bm{\mathrm{x}}]\bm{\mathrm{y}} for any pair $\bm{\mathrm{x}}$ and $\bm{\mathrm{y}}$ in a fashion akin to a sparse-matrix-vector product but in runtime proportional to the number of triangles in the graph.

This directly enables us to compute \mathchoice{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to3.24023pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to2.31444pt{}}}\hss}}}\bm{{T}}[\bm{\mathrm{x}}_{i-1}]\bm{\mathrm{y}}_{i-1}. To compute the entries in $\bm{{D}}^{-1}_{A_{i}}$, note that \mathchoice{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to3.24023pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to2.31444pt{}}}\hss}}}\bm{{T}}[\bm{\mathrm{x}}] is a symmetric matrix because it can be written as a sum of symmetric matrices (since \mathchoice{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to3.24023pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to2.31444pt{}}}\hss}}}\bm{{T}} is fully symmetric in all permutations). Thus, the row-sums of $\bm{{A}}_{i}$ are the vertex-degrees we need to build $\bm{{D}}^{-1}_{A_{i}}$. Let $\bm{\mathrm{e}}$ be the vector of all ones; these row sums are computed as \bm{{A}}_{i}\bm{\mathrm{e}}=\mathchoice{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to3.24023pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to2.31444pt{}}}\hss}}}\bm{{T}}[\bm{\mathrm{x}}_{i}]\bm{\mathrm{e}}+\bm{{A}}\bm{\mathrm{e}}. Since $\bm{{A}}$ is not changing, we only need to compute the column sums of \mathchoice{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to3.24023pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to2.31444pt{}}}\hss}}}\bm{{T}}[\bm{\mathrm{x}}_{i}]\bm{\mathrm{e}} at each iteration. Again, we can use an implicit tensor-vector-vector product operation to compute the column sums. And thus, all operations involving the tensor \mathchoice{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to4.6289pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to3.24023pt{}}}\hss}}}{\hbox to0.0pt{{\underline{{\hbox to2.31444pt{}}}\hss}}}\bm{{T}} are linear in terms of the number of triangles in the network, and we use a fast routine to iterate through triangles in a graph. For ease of reuse, we provide the code for TRPR at https://github.com/nassarhuda/pairseed/blob/master/trpr.jl. We experimentally validate the running time of TRPR on a preferential attachement graph while varying the size of the graph. We specifically use the generalized preferential attachment model Avin et al. [2015] that generalizes the classical preferential attachment model Newman [2001]. In this experiment we vary the edge addition probability $p_{e}$, and allow the node addition probability to be $1-p_{e}$. Figure 3.3 shows the running time in seconds and empirically verifies that TRPR is a fast method when implemented efficiently and thus, is scalable to large graphs.

††margin:

Figure 3: Time in seconds as we run TRPR for 10 iterations on generalized preferential attachement graphs as we vary the size of the network and the edge addition probability.

A weighted version of TRPR.  Although TRPR introduces higher weights to edges participating in many triangles by forming a new adjacency matrix $\hat{\bm{{X}}}+\bm{{A}}$, these weights are often dominated by the weights in the adjacency matrix $\bm{{A}}$. To give a fair contribution to these edges, we introduce a scalar multiple to $\hat{\bm{{X}}}$. A straightforward scalar we choose is $\gamma=\text{sum}(\bm{{A}})/\text{sum}(\hat{\bm{{X}}})$. This scalar will guarantee that the sum of weights in $\bm{{A}}$ and $\gamma\hat{\bm{{X}}}$ are equal. We present the idealized algorithm of the weighted version of TRPR in Algorithm 2.

††margin:

Figure 4: 1-norm convergence of TRPR on 4 datasets used in the experiments section. These figures show that TRPR converges experimentally. ††margin:

Figure 5: Spearman’s rank correlation coefficient and the Kendall rank correlation coefficient between two consecutive iterates from TRPR. The solid plots show the consecutive correlation values when truncating the vectors to take the top 100 nodes, and the dashed lines compare the orderings in the full vectors. The vertical red line represents the $10^{th}$ iterate. The text in the figures is the correlation between the $10^{th}$ iterate and the $200^{th}$ iterate. These correlations support our choice of 10 iterations in the experiments involving TRPR.

Convergence of TRPR.  Convergence of this type of nonlinear system of equations is theoretically delicate with bounds that are often insufficient for practice Benson et al. [2017]. Empirically, we observe that the iterations converge. However, absent a robust theory, this method is only run for a small and fixed number of iterations (10). This will produce a unique deterministic and reproducible set of scores that locally capture the influence of both the graph and the reinforced triangles. In Figure 3.3, we show the 1-norm difference decay from two consecutive iterates from TRPR on 4 datasets used in the experiments section. Figure 3.3 shows that the method converges experimentally. Even though the norm convergence seems to happen after around 100 iterations in these datasets, the ordering of nodes in these vectors does not change much after a few iterations. We run another experiment to study the ordering of the nodes from every iteration and notice that the order does not change much after just a few iterations. In Figure 3.3, we show the Spearman’s rank correlation coefficient and the Kendall rank correlation coefficient between two consecutive iterates from TRPR on the same four graphs used in Figure 3.3. We notice that after a few iterations (10) the orderings of the vectors no longer change, and especially the order of the top 100 nodes does not change (solid lines in the plot in Figure 3.3).

3.4  Extensions of single-seeded PageRank

We also use the single seeded PageRank solution of each endpoint of the edge we are interested in predicting links to and produce two more metrics for relating an edge to a node. Denote $\bm{\mathrm{x}}_{u}$, and $\bm{\mathrm{x}}_{v}$ to be the seeded PageRank solutions for nodes $u$ and $v$ respectively. Then, we define MAX and MUL as follows.

[TABLE]

4   Experimental Setup

We now perform a series of experiments on synthetic as well as real-world graphs from a variety of disciplines, including online social networks, communication networks, and biological interaction networks. We also include experiments for static networks as well as a temporal network. For evaluation, we use the Success Probability (SP) measure, which we define for one experiment as follows:

[TABLE]

Note that the top $k$ predictions is the set of $k$ nodes that are not connected to either end point of the seed edge with the highest scores. For our experiments, we will have training data and validation data, and the ground truth nodes that should be connected to an edge from the training data can be deduced from the validation data. For each training dataset, we run $500$ random experiments, where we try to predict links to $500$ randomly chosen edges (we call them seed edges). For each experiment, an SP value ([math] or $1$) is computed, and the overall score is the mean value over all the experiments. The main choice for this measure in contrast to the area under ROC curve (AUC score) measure for instance, is the small number of nodes that we often want to recover. For a given edge $(u,v)$ in the training data, the validation data must have the edges $(u,w)$ and $(v,w)$ for $w$ to be considered a correct ground truth node for recovery. In subsequent sections, we will see that the number of nodes that satisfy this property in the validation data is often small ($1$ in most instances), and thus a measure such as the AUC score does not fully capture the performance of our methods.

4.1  Leave One Edge’s Triangles Out (LOETO)

The LOETO experiments are akin to the leave-$p$-out cross validation metric, in the sense that we will use $p$ edges as a validation set and the remaining edges of the network as a training set; here, $p=(2$ $\times$ number of nodes that form a triangle with a randomly chosen edge$)$. An experimental trial in this setting is designed as follows. Randomly pick an edge in the graph (call it the seed edge) and find all the wedges (path of length 2) that form a triangle with this edge. Next, drop all these wedges and place them in the validation set. Figure 4.2 visualizes this experiment. The graph used will be the one in panel B of Figure 4.2 (the grey dashed edges no longer appear in the network and the goal is to recover the connections with the green nodes in the graph). We then use the pairwise link prediction methods on the seed edge, which produces an ordering on the nodes, and given this ordering, we compute the success probability. Since this method leaves a big portion of the graph in the training data, we compute its Success Probability with top $k=5$.

4.2  Hold-out cross validation

The hold-out cross validation method that keeps a certain percentage of the data as training set and the remaining set as validation is a standard way of evaluating the classical link prediction problem. In this setup, for a given network, we remove $30\%$ of the edges and label them as validation data, and use the remaing $70\%$ as training data to make predictions. Then, for random edges in the training data (seed edges), we use the pairwise link prediction methods to predict which nodes will form triangles with each edge that is selected. Again, for a given edge, each method produces a similarity score on all nodes, and we use the ordering of the nodes induced by the scores to calculate the Success Probability with top $k$ values $=5,25$.

††margin:

Figure 6: Illustration of the Leave One Edge’s Triangles Out (LOETO) experiment. For a given graph (subfigure A), randomly pick an edge (red edge in subfigure B) and remove all edges that form a triangle with it (dashed gray lines in subfigure B). Run all our methods on this new graph. The nodes to predict are the green nodes.

We also perform a similar experiment on temporal networks with timestamps on the edge arrivals. In this scenario, the dropped $30\%$ edges are not chosen at random. Instead, we split the data into training and test sets based on the time—the first $70\%$ of the edges to appear in time are the training data and the remaining $30\%$ are the test data.

In this set of experiments, we perform one more processing step to guarantee that the network we will use for training is connected. If the network is disconnected, we extract the largest connected component.

4.3  Summary of methods and parameter settings

Finally, we summarize all of the methods that we use for pairwise link prediction.

• ·

Pairseed: This is our method described in Section 3.2. We use the implementation from MatrixNetworks.jl Nassar and Gleich [2018] with $\alpha=0.85$. This implementation solves the linear system until convergence to machine precision.

• ·

TRPR: This is our method described in Section 3.3. We use $\alpha=0.85$ and number of iterations $n=10$.

• ·

TRPRW: This is the modified weighted version of the TRPR algorithm described in Section 3.3 as well. We use $\alpha=0.85$ and number of iterations $n=10$.

• ·

MUL, MAX: These are the methods from Section 3.4 that extend the single-seeded PageRank solutions. We use the same implementation used by Pairseed, with $\alpha=0.85$.

• ·

AA, PA, JS: For a seed edge, we compute the generalized Adamic-Adar, Preferential Attachment, and Jaccard similarity scores, respectively (as presented in Section 3.1) between the seed edge and all remaining nodes in the graph.

• ·

AA–MUL, AA–MAX, JS–MUL, JS–MAX: These are the methods from Section 3.1, and they use the single node similarity from both endpoints of a seed edge to compute a new measure of similarity.

5   Pairwise link prediction results

In all of the results in this section, we report the success probability from our predictions over $500$ random experiments. We use seven real-world graphs from different disciplines in this section and give a summary of their statistics in Table 1. We also use a synthetic graph generated from the generalized preferential attachment model (GPA) Avin et al. [2015].

Synthetic graph.  Generalized Preferential Attachment (GPA) Newman [2001] is a synthetic graph generation model that generalizes the classical prefernetial attachment model to allow for the addition of new components at each step of the algorithm. For our experiments, we generate a graph with $5000$ nodes and allow the event of node addition with probability $1/2$, and we allow the event of edge addition with probability $1/2$. The starting graph structure is a clique of size $5$. At each step of the graph generation process, an edge or node is added by attaching proportionally to the degrees of the existing nodes.

Real world graphs.  We use various real world graphs to test our methods and provide statistics about them in Table 1. Penn94 and Caltech36 are online social networks from the Facebook100 collection of datasets Traud et al. [2011]. These two datasets are the biggest and smallest networks in terms of number of nodes respectively from this collection. Ch-Ch-Miner is a biological network of drug (chemical) interactions Wishart et al. [2017]; Stanford SNAP Group [2017]. P-P-Pathways is a biological network of physical interactions between proteins in humans Agrawal et al. [2018]. email is an email communication network Guimerà et al. [2003]. Finally, CollegeMsg Panzarasa et al. [2009] and email-EU Panzarasa et al. [2009] are temporal networks representing private messages (CollegeMsg) or emails (email-EU) between users in a network.

Results.  We show the results of all methods in Figures 7, 8, 9, and 10. Overall, we notice that the diffusion methods have more consistency in performance compared to local measures. For instance, AA-MUL — which is one of the best performers on some datasets (P-P-Pathways in top 25 and top 5 metrics) — drops to be one of the worst performers in the top 25 metric on the email dataset. PA is the best performer on the GPA model, but drops to be the worst performer on all other graphs. In cotrast, TRPRW performs best on the Ch-Ch-Miner and email datasets but never drops to be one of the worst methods on any of the datasets. Temporal graphs (CollegeMsg and Email-EU) both suffered from lower top $k$ scores as compared to static graphs, which suggests that our methods are possibly stronger in detecting missing links rather than future links. Upon further investigation on the temporal graphs, we found that most of the top $k$ predictions were at least two hops away from the seed edges. In the temporal data, these wedges (length-2 paths) did not close to form triangles and thus the prediction was incorrect according to the timestamped data. TRPRW seemed to improve the performance of TRPR in general but did not contribute an improvement on the email and GPA networks. Upon looking closely at these two networks, we found that the number of triangles is very small and thus using the unweighted TRPR version which is close in performance to Pairseed, is more ideal on datasets that do not contain many triangles.

6   Back to Standard Link Prediction

In this section we bring our attention back to the standard link prediction problem and show how the methods we presented in this paper can also be used to further enhance standard link prediction. We split our data in the same way to the previous experiments except that here we use an 80-20 split (often, keeping a higher percentage of the data in the training set produces higher quality results, but in the previous section, we needed to generate more data in the validation set so that we have higher chances of finding paths of length 2 to predict, and thus we increased the size of the testing data by 10 percent). Then, for the top 100 nodes with the largest degree in the training data, we perform different types of seeded PageRank diffusion for link prediction on these nodes. This choice of nodes serves the purpose of identifying nodes that have a higher chance of making connections in the test data. We measure performance in terms of Area Under the ROC curve (henceforth, AUC score). Our baseline is single-seeded PageRank.

Our results on pairwise link prediction suggest that multiple seeds with PageRank-like methods are effective for prediction. Here, we consider four different multiple-seeding strategies and compare them to single-seeded PageRank for the classical link prediction problem. We summarize the four new methods in Table 2. The methods sum, max, and star-seed are motivated by the double seeding idea used in the previous sections.

We use real-world networks from Section 5, and present our results in Figure 11. The scatter plots compare the AUC score of the neighborhood-based seeding methods to the AUC scores from single-seeded PageRank. These results suggest that neighborhood-based seeding is superior to single-seeded PageRank as a link prediction method.

7   Discussion and Future work

Having a reliable link prediction algorithm is a well-studied research topic due to its utility in many disciplines. Traditional link prediction methods aim to find pairs of nodes that are likely to form a link. Here, we have studied a higher-order version of the problem called pairwise link prediction where we predict nodes that are likely to form a triangle with an edge. We generalized local link-prediction methods and we developed two PageRank-based methods for this problem. These PageRank-based methods generally remained consistent in behavior on a variety of datasets. Using these results as inspiration, we then developed multiple-seeding strategies for PageRank in classical link prediction, which outperform their standard single-seeded counterparts.

TRPR (Triangle Reinforced PageRank) is our new principled method for the task of pairwise link prediction. We demontrated that TRPR is computationally efficient, and demonstarted the implementation details of TRPR can improve on the idealized algorithm by taking advantage of a triangle iterator that avoids building a tensor. We note that highly efficient implementations of our procedures are possible given their close relationships with traditional PageRank methods. Scaling to billions of nodes and edges is simply not a problem given current abilities to compute PageRank (e.g. Lofgren et al. [2016]), and especially that we have an existing routine to iterate through triangles in a graph quickly.

The space of higher-order prediction problems also has limitless sub-structure. An alternate problem is to predict an edge that is important when given a single node. In the future, we intend to extend this work to the latter scenario, and TRPR can be adapted for this purpose.

Acknowledgements

Supported by NSF IIS-1422918, IIS-1546488, CCF-1909528, NSF Center for Science of Information STC, CCF-0939370, NASA, Sloan Foundation, DARPA SIMPLEX, NSF DMS-1830274, ARO W911NF-19-1-0057, and ARO MURI.