Community detection over a heterogeneous population of non-aligned networks

TL;DR

This paper introduces a joint stochastic blockmodel for community detection across multiple heterogeneous, non-aligned graphs, leveraging cross-graph information to improve community estimation.

Contribution

It develops a novel joint SBM framework and an efficient spectral clustering method for community detection in unaligned, heterogeneous graphs.

Findings

Joint SBM outperforms separate SBMs in community estimation

Spectral clustering efficiently learns model parameters

Model validated on synthetic and real-world datasets

Abstract

Clustering and community detection with multiple graphs have typically focused on aligned graphs, where there is a mapping between nodes across the graphs (e.g., multi-view, multi-layer, temporal graphs). However, there are numerous application areas with multiple graphs that are only partially aligned, or even unaligned. These graphs are often drawn from the same population, with communities of potentially different sizes that exhibit similar structure. In this paper, we develop a joint stochastic blockmodel (Joint SBM) to estimate shared communities across sets of heterogeneous non-aligned graphs. We derive an efficient spectral clustering approach to learn the parameters of the joint SBM. We evaluate the model on both synthetic and real-world datasets and show that the joint model is able to exploit cross-graph information to better estimate the communities compared to learning separate SBMs on each individual graph.

Tables

Table 1. Table 1: Overall and Individual (mean and interquartile range) NMIs for Karnataka villages dataset.

Model	Overall	Individual
	NMI	NMI
JointSpec	0.134	0.6126 [0.211,1]
IsoSpec	$0.091$	$0.4995$ $[0.191,0.789]$
node2vec	$0.113$	$0.43807$ $[0.052,0.744]$

Table 2. Table 2: Top 3 3 3 words assigned to communities by each model and each side (pro and against government)

$𝑲$	JointSpec		IsoSpec
	Pro	Against	Pro	Against
1	naovaitergolpe	foradilma	naovaitergolpe	foradilma
	golpe	forapt	golpe	forapt
	dilmafica	dilma	foradilma	dilma
2	hora	janaiva	turno	arte
	galera	compartilhar	venceu	objetos
	democralica	faltam	anavilarino	apropriou
3	sociais	mito	rua	rua
	coxinha	elite	april	coxinha
	naonaors	compartilhar	continuar	elite
4	vai	lulanacadeia	dilmafica	impeachment
	brasil	lula	dilma	brasil
	povo	vai	forapt	lulanacadeia

Table 3. Table 3: Notation

$K$	number of communities
$𝚯$	$K\times K$ connectivity matrix
$V_n$	Nodes in graph $n$
$V$	${\bigcup }_{n=1}^N{V}_n$
$G_{nk}$	Nodes in graph $n$ and in community $k$
$G_{\cdot k}$	${\bigcup }_{n=1}^N{G}_{nk}$
${𝑨}_n$	$\left|V_n\right|\times \left|V_n\right|$ adjacency matrix of graph $n$
${𝑷}_n$	$\left|V_n\right|\times \left|V_n\right|$ edge probabilities matrix of graph $n$
${𝑿}_n$	$\left|V_n\right|\times K$ membership matrix of graph $n$
${\Delta }_n$	${\left({𝑿}_n^{\prime }{𝑿}_n\right)}^{1/2}$
${𝑼}_n$	$\left|V_n\right|\times K$ matrix of eigenvectors of $P_n$
$𝑨$	$\left|V\right|\times \left|V\right|$ block diagonal matrix of all ${𝑨}_n$
$𝑷$	$\left|V\right|\times \left|V\right|$ matrix of all edge probabilities
$𝑿$	$\left|V\right|\times K$ of all membership matrices stacked
$\Delta$	${\left({𝑿}^{\prime }𝑿\right)}^{1/2}$
$𝑼$	$\left|V\right|\times K$ matrix of ${𝑼}_n$s stacked
$\parallel \cdot \parallel$	Euclidean (vector) and spectral (matrix) norm
${\Vert M\Vert }_F$	Frobenius norm of matrix $M$
${\Vert M\Vert }_0$	The $l_0$-norm of matrix $M$

Equations

$$a_{ij}\sim \text{Bern}({\mathbf{X}}_i\mathbf{\Theta }{\mathbf{X}}_j^T)$$

$$a_{nij}\sim \text{Bern}({\mathbf{X}}_{ni}\mathbf{\Theta }{\mathbf{X}}_{nj}^T)$$

$${\mathbf{U}}_n{\mathbf{D}}_n{\mathbf{U}}_n^T={\mathbf{P}}_n$$

$${\mathbf{D}}_n={\tilde{\mathbf{D}}}_n,{\mathbf{U}}_n={\mathbf{X}}_n{\Delta }_n^{-1}{\mathbf{Z}}_n.$$

$$\left({\hat{\mathbf{X}}}_n,{\hat{\mathbf{W}}}_n\right)=\underset{\begin{array}{c}{\mathbf{X}}_n\in {\mathcal{M}}_{|V_n|,K}\\ {\mathbf{W}}_n\in {\mathbb{R}}^{K\times K}\end{array}}{\mathrm{argmin}}\Vert {\mathbf{X}}_n{\mathbf{W}}_n-{\hat{\mathbf{U}}}_n{\Vert }_F^2$$

$${\hat{\mathbf{\Theta }}}_n=$$

$$\mathbf{A}=\left[\begin{array}{ccccc}{\mathbf{A}}_1& \dots & 0& \dots & 0\\ ⋮& \ddots & ⋮& ⋮& ⋮\\ 0& \dots & {\mathbf{A}}_n& \dots & 0\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ 0& \dots & 0& \dots & {\mathbf{A}}_N\end{array}\right],\mathbf{P}=\left[\begin{array}{ccccc}{\mathbf{P}}_1& \dots & {\mathbf{P}}_{1n}& \dots & {\mathbf{P}}_{1N}\\ ⋮& \ddots & ⋮& ⋮& ⋮\\ {\mathbf{P}}_{n1}& \dots & {\mathbf{P}}_n& \dots & {\mathbf{P}}_{nN}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ {\mathbf{P}}_{N1}& \dots & {\mathbf{P}}_{Nn}& \dots & {\mathbf{P}}_N\end{array}\right].$$

$$\mathbf{U}\mathbf{D}{\mathbf{U}}^T=\mathbf{P}$$

$$\mathbf{D}=\tilde{\mathbf{D}},\mathbf{U}=\mathbf{X}{\Delta }^{-1}\mathbf{Z},$$

$${\mathbf{P}}_n={\left(\mathbf{P}\right)}_{n,n}={\left(\mathbf{U}\mathbf{D}{\mathbf{U}}^T\right)}_{n,n}={\mathbf{U}}_{n\ast }\mathbf{D}{\mathbf{U}}_{n\ast }^T.$$

$${\mathbf{U}}_{n\ast }\mathbf{D}{\mathbf{U}}_{n\ast }^T$$

$${\mathbf{U}}_{n\ast }\mathbf{D}{\mathbf{U}}_{n\ast }^T{\mathbf{U}}_n$$

$${\mathbf{U}}_{n\ast }\mathbf{D}({\mathbf{X}}_{n\ast }{\Delta }^{-1}\mathbf{Z}{)}^T{\mathbf{X}}_n{\Delta }_n^{-1}{\mathbf{Z}}_n$$

$${\mathbf{U}}_{n\ast }\mathbf{D}{\mathbf{Z}}^T{\Delta }^{-1}{\mathbf{X}}_{n\ast }^T{\mathbf{X}}_n{\Delta }_n^{-1}{\mathbf{Z}}_n$$

$${\mathbf{U}}_{n\ast }\mathbf{D}{\mathbf{Z}}^T{\Delta }^{-1}{\Delta }_n^2{\Delta }_n^{-1}{\mathbf{Z}}_n$$

$${\mathbf{U}}_{n\ast }\mathbf{D}={\mathbf{Q}}_n{\mathbf{Z}}_n^T{\Delta }_n^{-1}\Delta \mathbf{Z}={\mathbf{X}}_{n\ast }\mathbf{W}$$

$$\left(\hat{\mathbf{X}},\hat{\mathbf{W}}\right)=\underset{\begin{array}{c}\mathbf{X}\in {\mathcal{M}}_{|V|,K}\\ \mathbf{W}\in {\mathbb{R}}^{K\times K}\end{array}}{\mathrm{argmin}}\sum _n^N\Vert {\mathbf{X}}_n\mathbf{W}-{\mathbf{Q}}_n{\mathbf{Z}}_n^T{\Delta }_n^{-1}\Delta \mathbf{Z}{\Vert }_F^2$$

$$\underset{\begin{array}{c}\mathbf{X}\in {\mathcal{M}}_{|V|,K}\\ \mathbf{W}\in {\mathbb{R}}^{K\times K}\end{array}}{\mathrm{argmin}}\sum _{n=1}^N{\Vert a_n({\mathbf{X}}_n,\mathbf{W})+b_n({\mathbf{X}}_n)\Vert }_F^2,\text{where}$$

$$a_n({\mathbf{X}}_n,\mathbf{W}):=\left({\mathbf{X}}_n\mathbf{W}-{\mathbf{Q}}_n^{\ast }\right){\mathbf{Z}}^T{\Delta }^{-1}{\Delta }_n{\mathbf{Z}}_n,b_n({\mathbf{X}}_n):={\mathbf{Q}}_n^{\ast }\left({\mathbf{Z}}^T{\Delta }^{-1}{\Delta }_n{\mathbf{Z}}_n-\sqrt{\frac{|V_n|}{|V|}}{\mathbb{I}}_K\right)$$

$$\frac{1}{2}{\Vert a_n({\mathbf{X}}_n,\mathbf{W})+b_n({\mathbf{X}}_n)\Vert }_F^2$$

$$:=$$

$$\text{where }{\gamma }_n$$

$$=\text{tr}\left({\Delta }_n^2{\Delta }^{-2}\right)=\sum _{m=1}^K\frac{|G_{nm}|}{|G_{\cdot m}|}.$$

$$\left(\hat{\mathbf{X}},\hat{\mathbf{W}}\right)=\underset{\begin{array}{c}\mathbf{X}\in {\mathcal{M}}_{|V|,K}\\ \mathbf{W}\in {\mathbb{R}}^{K\times K}\end{array}}{\mathrm{argmin}}\sum _n^N{\eta }_n({\mathbf{X}}_n,\mathbf{W})$$

$$\sum _n^{N}2{\mathbf{X}}_n^T({\mathbf{X}}_n\hat{\mathbf{W}}-{\mathbf{Q}}_n^{\ast }){\gamma }_n=0$$

$$\hat{\mathbf{W}}={\left[\sum _n^N{\mathbf{X}}_n^T{\mathbf{X}}_n{\gamma }_n\right]}^{-1}\sum _n^N{\mathbf{X}}_n^T{\mathbf{Q}}_n^{\ast }{\gamma }_n$$

$${\omega }_{ni}(k)={\Vert {\mathbf{W}}_k-{\mathbf{Q}}_{ni}^{\ast }\Vert }^2\text{tr}\left({\tilde{\Delta }}_{ni}^2(k)\right)+{\Vert \left|{\mathbf{Q}}_{ni}^{\ast }\right|\left({\tilde{\Delta }}_{ni}(k)+\sqrt{\frac{|V_n|}{|V|}}{\mathbb{I}}_K\right)\Vert }^2$$

$${\tilde{\Delta }}_{ni}(k)={\left[({\Delta }^2-\text{diag}(H_l)+\text{diag}(H_k))\right]}^{-1/2}\times {\left[({\Delta }_n^2-\text{diag}(H_l)+\text{diag}(H_k))\right]}^{1/2}$$

$${\mathbf{X}}_{ni}=$$

$$\mathbb{E}\left[{\mathbf{Z}}_n^T{\Delta }_n^{-1}\Delta \mathbf{Z}\right]=\sqrt{|V_n{|}^{-1}|V|}$$

$$\mathbb{E}\left[{\eta }_n({\mathbf{X}}_n,\mathbf{W})\right]={\Vert {\mathbf{X}}_n\mathbf{W}-{\mathbf{Q}}_n^{\ast }\Vert }_F^2\frac{|V_n|K}{|V|}+{\Vert 2\left|{\mathbf{Q}}_n^{\ast }\right|\sqrt{\frac{|V_n|}{|V|}}\Vert }_F^2$$

$$\mathbb{E}[{\eta }_n({\mathbf{X}}_n,\mathbf{W})]\propto \Vert {\mathbf{X}}_n{\mathbf{W}}_n-{\hat{\mathbf{U}}}_n{\Vert }_F^2$$

$$\begin{array}{r}{\mathbf{\pi }}_n|\alpha \sim \text{Dir}\left(K,\frac{1}{\alpha K}\right),\end{array}$$

$$a_{nij}|{\mathbf{X}}_{ni},{\mathbf{X}}_{nj},{\left({\theta }_{kl}\right)}_{k=1,l=1}^{K,K}$$

Full text

1 Introduction

Interest in the statistical analysis of network data has grown rapidly over the last few years, with applications such as link prediction [Liben-Nowell and Kleinberg, 2007, Wang et al., 2015], collaborative filtering [Sarwar et al., 2001, Linden et al., 2003, Koren and Bell, 2015], community detection [Fortunato, 2010, Aicher et al., 2014, Newman, 2016] etc. In the area of community detection, most work has focused on single large graphs. Efforts to move beyond this to consider scenarios with multiple graphs, such as multi-view clustering [Cozzo et al., 2013], multi-layer community detection [Mucha et al., 2010, Zhou et al., 2017], and temporal clustering [Von Landesberger et al., 2016]. However, these efforts assume that the graphs are aligned, with a known correspondence or mapping between nodes in each graph. Here, the multiple graphs provide different sources of information about the same set of nodes. Applications of aligned graphs include graphs evolving over time [Peixoto, 2015, Sarkar et al., 2012, Charlin et al., 2015, Durante and Dunson, 2014]; as well as independent observations [Ginestet et al., 2017, Asta and Shalizi, 2015, Durante et al., 2015, 2018].

Such methods are not applicable to multiple non-aligned graphs, which occur in fields like biology (e.g., brain networks, fungi networks, protein networks), social media (e.g., social networks, word co-occurrences), and many others. In these settings, graph instances are often drawn from the same population, with limited, or often no, correspondence between the nodes across graphs (i.e., the nodes have similar behavior but represent different entities). Clustering and community detection methods for this scenario are relatively under-explored. While one could cluster each graph separately, pooling information across graphs can improve estimation, particularly for graphs with sparse connectivity or imbalanced community sizes.

Here, we focus on the problem of community detection across a set of non-aligned graphs of varying size. Formally, we are given a set of $N$ graphs, $\bm{\Omega}=\{\bm{A}_{1},\dotsc,\bm{A}_{N}\}$, where the $n$th graph is represented by its adjacency matrix defined as $\bm{A}_{n}\in\{0,1\}^{|V_{n}|\times|V_{n}|}$ where ${V}_{n}$ is the set of nodes. Write ${E}_{n}$ for the edges. We do not require the set $V_{n}$ to be shared across different graphs, however we assume they belong to a common set of $K$ communities. Our goal is to identify the $K$ communities underlying $\bm{\Omega}$. For instance, consider villages represented by social graphs where nodes represent individuals in a village and edges represent relationships between them. Although the people are different in each village and the sizes of villages vary, personal characteristics may impact the propensity of having some type of relationship, e.g. a younger individual is more willing to connect with other younger individuals, or an influential person such as a priest might be expected to have more ties. Indeed, there are a vast number of factors, both observed and latent, shared among people across villages, that might influence relationship and community formation.

Our approach builds on the popular stochastic block model (SBM). SBMs have been studied extensively for single graph domains [Rohe et al., 2011, Lei et al., 2015, Sarkar et al., 2015], and have proven to be highly effective on real world problems [Airoldi et al., 2008, Karrer and Newman, 2011]. A simple approach to apply SBMs in our setting is to separately estimate the SBM parameters per graph, and then attempt to find a correspondence between the estimated probability structures to determine a single global community structure. We call this process Isolated SBM and show it is only reasonably accurate when each community is well represented in each graph. Moreover, the process of finding a correspondence among the estimated probabilities has $\mathcal{O}(NK^{2})$ complexity, which is only feasible when $N$ is small and $K$ is very small.

We propose a joint SBM model, where the key issue is to estimate a joint connectivity structure among the communities in each graph while allowing the number and sizes of clusters to vary across graphs. To estimate the parameters of the model, we show how the individual graph adjacency matrix eigendecompositions relate to the decomposition of the whole dataset $\bm{\Omega}$, and then derive an efficient spectral clustering approach to optimize the joint model. Our learning algorithm has complexity $\mathcal{O}(|V|K)$ per iteration where $|V|$ is total number of nodes across graphs. Notably, our approach is more efficient than Isolated SBM because it does not need the $\mathcal{O}(NK^{2})$ step to determine a correspondence between the separate models across graphs. We evaluate our Joint SBM on synthetic and real data, comparing the results to the Isolated SBM and an alternative approach based on clustering graph embeddings. The results show that our joint model is able to more accurately recover the community structure, particularly in scenarios where graphs are highly heterogeneous.

2 Joint SBM for multiple graphs

2.1 Single graph SBM

The stochastic blockmodel (SBM) [Holland et al., 1983, Wasserman and Anderson, 1987] can be viewed as a mixture of Erdös-Rényi graphs [Erdos and Renyi, 1959], and for a single graph $\bm{A}$ with $K$ communities, is defined by a $(|V_{n}|\times K)$ membership matrix $\bm{X}$, and a connectivity matrix $\bm{\Theta}\in[0,1]^{K\times K}$. Here $\bm{X}[i,k]=1$ if the $i$-th node belongs to community $k$, and equals [math] otherwise. $\bm{\Theta}[k,l]=\theta_{kl}$ is the probability of an edge between nodes from communities $k$ and $l$. Then, a graph represented by adjacency matrix $\bm{A}$ is generated as

[TABLE]

Note that $a_{ij}=a_{ji}\ \forall i,j$, and since we do not consider self-edges, $a_{ii}=0\ \forall i$. SBMs have traditionally been applied in settings with a single graph. Here, theoretical properties like consistency and goodness-of-fit are well understood, and efficient polynomial-time algorithms with theoretical guarantees have been proposed for learning and inference. However, there is little work for the situation with multiple observed graphs with shared statistical properties.

2.2 Multi-graph joint SBM

To address this, we consider an extension of the SBM in Equation (1). Our model does not require vertices to be aligned across graphs, nor does it require different graphs to have the same number of vertices. Vertices from all graphs belong to one of shared set of groups, with membership of graph $n$ represented by a membership matrix $\bm{X}_{n}$. Edge-probabilities between nodes are determined by a global connectivity matrix $\bm{\Theta}$ shared by all graphs. For notational convenience, we will refer to the set of stacked $\bm{X}_{n}$ matrices as the full membership matrix $\bm{X}$, with $\bm{X}_{ni}$ one-hot membership vector of the $i$-th node of graph $n$. The overall generative process assuming $K$ blocks/clusters/communities is

[TABLE]

where again, $a_{nij}$ is $ij$-th cell of the adjacency matrix $\bm{A}_{n}$. Note that $\bm{X}_{n}$ is a $(|V_{n}|\times K)$ binary matrix, and $\bm{\Theta}$ is a ${K\times K}$ matrix of probabilities. This model can easily be extended to edges with weights (replacing the Bernoulli distribution with some other distribution) or to include covariates (e.g. through another layer of coefficients relating covariates with membership or edge probabilities).

3 Inference via spectral clustering

Having specified our model, we provide a spectral clustering algorithm to identify community structure and edge probabilities from graph data $\bm{A}_{1},\bm{A}_{2},\ldots,\bm{A}_{N}$.

3.1 Spectral clustering for a single graph

First, we recall the spectral clustering method to learn SBMs for a single graph $\bm{A}_{n}$ [von Luxburg, 2007, Lei et al., 2015]. Let $\bm{P}_{n}$ refer to the edge probability matrix of graph $n$ under an SBM, where $\bm{P}_{n}=\bm{X}_{n}\mathbf{\Theta}\bm{X}_{n}^{T}$ (Equation (1)). Since we do not consider self-loops, $\mathbb{E}[\bm{A}_{n}]=\bm{P}_{n}-\text{diag}(\bm{P}_{n})$. Further, write the eigendecomposition of $\bm{P}_{n}$ as $\bm{P}_{n}=\bm{U}_{n}\bm{D}_{n}\bm{U}_{n}^{T}$. Here, $\bm{U}_{n}$ is a $(|V_{n}|\times K)$ matrix of eigenvectors related to the $K$ largest absolute eigenvalues and $\bm{D}_{n}$ is a $K\times K$ diagonal matrix with the $K$ non-zero eigenvalues of $\bm{P}_{n}$. Let $|G_{nk}|$ refer to the number of nodes that are members of cluster $k$, and $\Delta_{n}=\left(\bm{X}_{n}^{T}\bm{X}_{n}\right)^{1/2}$ define a $K\times K$ diagonal matrix with entries $\sqrt{|G_{nk}|}$. Define $\bm{Z}_{n}\tilde{\bm{D}}_{n}\bm{Z}_{n}^{T}$ as the eigendecomposition of $\Delta_{n}\mathbf{\Theta}\Delta_{n}$. Then

[TABLE]

Since $\bm{D}_{n}$ and $\tilde{\bm{D}}_{n}$ are both diagonal, and $\bm{X}_{n}\Delta_{n}^{-1}$ and $\bm{Z}_{n}$ are both orthonormal,

[TABLE]

In practice, we use the observed adjacency matrix $\bm{A}_{n}$ as a proxy for $\bm{P}_{n}$, and replace $\bm{U}_{n}$ in Eqs. (3) and (4) with $\widehat{\bm{U}}_{n}$ calculated from the eigendecomposition $\bm{A}_{n}=\widehat{\bm{U}}_{n}\widehat{\bm{D}}_{n}\widehat{\bm{U}}_{n}^{T}$. Finally we note that each row of $\bm{X}_{n}$ has only one non-zero element, indicating which group that node belongs to. Thus, as in Lei et al. [2015], we can use k-means clustering to recover $\bm{X}_{n}$ and $\bm{W}_{n}=\Delta_{n}^{-1}\bm{Z}_{n}$ from $\widehat{\bm{U}_{n}}$:

[TABLE]

Given a solution $\left(\widehat{\bm{X}}_{n},\widehat{\bm{W}}_{n}\right)$ from Equation (5), we can estimate $\bm{\Theta}$ from the cluster memberships $\widehat{\bm{X}}_{n}$ as:

[TABLE]

where $\widehat{\bm{S}}_{n}=\widehat{\Delta}_{n}^{-2}\widehat{\bm{X}}_{n}^{T}\bm{A}_{n}\widehat{\bm{X}}_{n}\widehat{\Delta}_{n}^{-2}$ and $\widehat{\Delta}_{n}^{2}=\widehat{\bm{X}}_{n}^{T}\widehat{\bm{X}}_{n}$. If we assume the true membership matrix $\bm{X}$ is known, then the estimator $\widehat{\bm{\Theta}}$ in Eq. (6) is unbiased for the true $\bm{\Theta}$. See Appendix 9.2 for details of derivation and unbiasedness.

3.2 Naive spectral clustering for multiple graphs

Given multiple unaligned graphs, the procedure above can be applied to each graph, returning a set of $\bm{X}_{n}$’s and $\bm{\Theta}_{n}$’s, one for each graph. The complexity of this is $\mathcal{O}(N\phi+|V|K)$, where $\phi$ refers to the complexity of eigen decomposition on a single graph (typically $\mathcal{O}(|E|)$ for sparse graphs Pan and Chen [1999], Golub and Van Loan [2012]). However, this does not recognize that a single $\bm{\Theta}$ is shared across all graphs. Estimating a global $\bm{\Theta}$ from the individual $\bm{\Theta}_{n}$s requires an alignment step, to determine a mapping among the $\bm{\Theta}_{n}$s. The complexity of determining the alignment is $\mathcal{O}(NK^{2})$ and is a two-stage procedure that results in loss of statistical efficiency, especially in settings with heterogenerous, imbalanced graphs. We refer to this approach as Isolated SBM. We propose a novel algorithm to get around these issues by understanding how each graph relates to the global structure.

3.3 Joint spectral clustering for multiple graphs

Let $|V|=\sum_{n}|V_{n}|$, where $|V_{n}|$ refers to the number of nodes in $A_{n}$. Consider $|V|\times|V|$ block diagonal matrix $\bm{A}$ representing the whole dataset of adjacency matrices, and define an associated probability matrix $\bm{P}$:

[TABLE]

Write the membership-probability decomposition of $\bm{P}$ as $\bm{P}=\bm{X}\mathbf{\Theta}\bm{X}^{T}$, here, $\bm{X}$ is the stacked $|V|\times K$ matrix of the $\bm{X}_{n}$’s for all graphs, and $\bm{\Theta}$ a $K\times K$ matrix of edge-probabilities among groups. Note that for $i\neq j$, $\bm{P}_{ij}$ includes edge-probabilities between nodes in different graphs, something we cannot observe. As before, define $\Delta=\left(\bm{X}^{T}\bm{X}\right)^{1/2}$, and the eigendecomposition of $\bm{P}$ gives

[TABLE]

with $\bm{Z}\tilde{\bm{D}}\bm{Z}^{T}$ corresponding to the eigendecomposition of $\Delta\mathbf{\Theta}\Delta$. Thus, similar to the single graph case

[TABLE]

Note that $\bm{D}$ is still a $K\times K$ diagonal matrix. Let $\bm{U}_{n*}=\bm{X}_{n*}\Delta^{-1}\bm{Z}$ refer to the subset of $\bm{U}$ corresponding to the nodes in graph $n$, and define $\bm{X}_{n*}$ similarly. Note that $\bm{X}_{n*}=\bm{X}_{n}$, though $\bm{U}_{n*}$ differs from $\bm{U}_{n}$ of Eq. (4). By selecting the decomposition related to graph $n$, we have,

[TABLE]

From Eq.(3), $\bm{U}_{n*}\bm{D}\bm{U}_{n*}^{T}=\bm{U}_{n}\bm{D}_{n}\bm{U}_{n}^{T}$. If we let $\bm{Q}_{n}:=\bm{U}_{n}\bm{D}_{n}$,

[TABLE]

From Eq. (7), we then have

[TABLE]

where $\bm{W}:=\Delta^{-1}\bm{Z}\bm{D}$. Contrast this with the single graph, which from Eq.(4), gives $\bm{U}_{n}\bm{D}_{n}=\bm{Q}_{n}=\bm{X}_{n}\Delta_{n}^{-1}\bm{Z}_{n}\bm{D}_{n}$.

If we can estimate the middle (or right) term in Eq. (10) from data, we can solve for joint community assignments:

[TABLE]

While we can estimate $\bm{Q}_{n}$ from the data (Equation (9)), we cannot estimate $\bm{Z}_{n}$ or $\bm{Z}$ trivially. Instead, we will optimize an upper bound of a transformation of Equation (11).

Lemma 3.1**.**

The optimization in Eq. (11) is equivalent to

[TABLE]

[TABLE]

Proof.

See Appendix 9.3. ∎

Next we derive a bound for Eq. (12) using the triangle inequality and sub-multiplicative norm property. Let $|\bm{M}|$ be the element-wise absolute values of matrix $\bm{M}$. Then

Lemma 3.2**.**

The following inequality holds

[TABLE]

[TABLE]

Proof.

See Appendix 9.4 ∎

Summary:

We can now optimize the bound on Eq. (12)

[TABLE]

The terms $\widetilde{a}_{n}(\cdot)$ and $\widetilde{b}_{n}(\cdot)$ are weighted sums of squares of $\bm{Q}_{n}$. However, we center each $\bm{Q}_{n}$ at the global weighted mean $\bm{W}$ in $\widetilde{a}_{n}(\cdot)$. The term $\gamma_{n}$ controls the importance of the global parameter $\bm{W}$ in each graph. Thus, $\gamma_{n}$ downweights the effect of $\bm{W}$ in small graphs and in graphs with highly underrepresented communities. Intuitively, the term $\widetilde{a}_{n}(\cdot)$ is assigning nodes to clusters assuming $\bm{Z}_{n}^{T}\Delta_{n}^{-1}\Delta\bm{Z}=\sqrt{|V|/|V_{n}|}\mathbb{I}_{K}$, and $\widetilde{b}_{n}(\cdot)$ accounts for the distance between a given graph and the global distribution of nodes over clusters.

Optimization:

We optimize Equation (15) using a heuristic inspired by Lloyd’s algorithm for k-means. This involves iterating two steps: (1) compute the means $\bm{W}$ given observations in each cluster $\bm{X}$; (2) assign observations to clusters $\bm{X}$ given means $\bm{W}$:

Compute the means: We update $\bm{W}$ by minimizing $\eta_{n}(\bm{X_{n}},\bm{W})$ for a given $\bm{X}$, i.e. $\sum_{n}^{N}\nabla_{{}_{\bm{W}}}\eta_{n}(\bm{X}_{n},\bm{W})=0$. Note that the $\widetilde{b}_{n}$ does not involve $\bm{W}$ so it is dropped,

[TABLE] 2. 2.

Assign nodes to communities: We assign each node to the cluster that minimizes Eq. (15). Accordingly, define $\omega_{ni}(k)$ as the distance of node $i$ to cluster $k$:

[TABLE]

$\bm{W}_{k}$ is the $k$-th row of $\bm{W}$ and $\widetilde{\Delta}_{ni}(k))$ is the value of $\Delta^{-1}\Delta_{n}$ if node $i$ is placed in cluster $k$. Precisely, say node $i$ is currently in cluster $l$, then we have

[TABLE]

where $H_{l}$ is a size $K$ one-hot vector at position $l$. Then

[TABLE]

Algorithm:

Algorithm 1 outlines the overall procedure for learning the Joint SBM. The complexity is $\mathcal{O}(N\phi+|V|K)$. Recall that $\phi$ refers to the complexity of eigen decomposition on a single graph. Note that our derived objective does not require decomposition of the full graph $\bm{A}$, instead decomposing each individual graph and then using the results to jointly estimate $\bm{X}$ and $\bm{W}$. Notice that the extra $\mathcal{O}(NK^{2})$ for alignment in the Isolated SBM is not needed here. Given the cluster assignments $\bm{X}$, we can easily estimate the cluster edge probabilities $\bm{\Theta}$ (see Eq.(6)).

4 Comparing Joint SBM to Isolated SBM

Both Joint SBM and Isolated SBM use the eigen decompositions of the individual $N$ graphs, and are closely related. The biggest difference is the $\bm{Z}_{n}^{T}\Delta_{n}^{-1}\Delta\bm{Z}$ term in the definition of $\widetilde{a}_{n}(\bm{X}_{n},\bm{W})$ (Eq. (13)), which, intuitively, normalizes each individual graph eigenvector based on the distribution of nodes over the clusters in the graph. If the graphs are balanced, i.e., they have roughly the same proportion of nodes over clusters, then $\bm{Z}_{n}^{T}\Delta_{n}^{-1}\Delta\bm{Z}\approx\sqrt{|V_{n}|^{-1}|V|}$ which does not depend on the cluster sizes. Lemma 20 formalizes this (See Appendix 9.5 for proof):

Lemma 4.1**.**

Let the pair $(\bm{X},\bm{\Theta})$ represent a joint-SBM with $K$ communities for $N$ graphs, where $\bm{X}$ is the stacked membership matrix over the $N$ graphs and $\bm{\Theta}$ is full rank. The size of each graph $|V_{n}|$ may vary, but assume the graphs are balanced in expectation in terms of communities, i.e., assume the same distribution of cluster membership for all graphs: $\bm{X}_{ni}\stackrel{{\scriptstyle iid}}{{\sim}}\text{Mult}(\bm{\zeta})$ for all $n\in[1,...,N]$ and $i\in[1,...,|V_{n}|]$. Then,

[TABLE]

For cases where we expect Lemma 20 to be true, we have:

[TABLE]

Now, we expect the RHS of Equation (15) to depend only on $\widetilde{a}_{n}(\bm{X}_{n},\bm{W})$, since $\widetilde{b}_{n}(\cdot)$ no longer depends on $\bm{X}_{n}$. If the graphs are also of the same size, then for each graph, the objective function of Joint SBM is equivalent to that of Isolated SBM. Lemma 21 formalizes this (See Appendix 9.6 for proof):

Lemma 4.2**.**

Let the pair $(\bm{X},\bm{\Theta})$ represent a joint-SBM with $K$ communities for $N$ graphs. If the sizes of the graphs are equal, i.e., $|V_{n}|=|V|/N$ and the graphs are balanced in expectation in terms of communities, i.e., assume the same distribution of cluster membership for all graphs: $\bm{X}_{ni}\stackrel{{\scriptstyle iid}}{{\sim}}\text{Mult}(\bm{\zeta})$ for all $n\in[1,...,N]$ and $i\in[1,...,|V_{n}|]$. Then,

[TABLE]

Lemma 21 illustrates the scenario when clustering the nodes of each graph individually (Isolated) and jointly have the same solution (i.e., based on optimizing $\bm{Q}_{n}$). However, this is only true with respect to the node assignments for each individual graph. If we look to the global assignments, the Isolated and the Joint model are expected to be the same only up to $N\!-\!1$ permutations of the community labels. Thus, for a good global clustering result, the Isolated model needs an extra $\mathcal{O}(NK^{2})$ step to realign the estimated $\bm{X}_{n}$s, which can also introduce additional error. If the data does not have graphs of the same size or the distribution of clusters varies across graphs, then the Isolated model will likely miss some blocks on each graph. Our joint method avoids these issues by using pooled information across the graphs to improve estimation. See Appendix 9.7 for a toy data example. Appendix 9.8 also includes a discussion of consistency.

5 Related work

Community detection in graphs has seen a lot of recent attention. We focus on extensions to multiple graphs, emphasizing two relevant directions: heterogeneity across communities and nonaligned graphs. For the former, Karrer and Newman [2011] propose a degree corrected SBM to account for heterogeneity inside a community, though Gulikers et al. [2017] showed that this fails to retrieve true communities in a high heterogeneous setting. Ali and Couillet [2017] introduce a normalized Laplacian form that account for high heterogeneous scenarios, however this comes at a high computational cost. Mucha et al. [2010] work with multiplex networks for time-dependent data, in their case each edge has multiple layers (attributes) which can be viewed as multiple aligned graphs. Our work is in a different domain, we work with non aligned networks.

More generally, methodology for multiple graphs can be divided into geometric  Ginestet et al. [2017], Asta and Shalizi [2015] and model-based  Durante et al. [2018], Duvenaud et al. [2015] approaches. The first approach seeks to characterize graphs topologically and explore hyperspace measures. Ginestet et al. [2017] introduces a geometric characterization of the network using the so-called Fréchet mean while their approaces are mathematically elegant, they are substantially less flexible than our work. The second approach aims to embed graph to a lower dimension space without oversimplifying the problem by making use of latent models. Durante et al. [2018], Gomes et al. [2018] proposed a mixture of latent space model to perform hypothesis testing on population of binary and weighted aligned graphs, respectively. On the other hand, [Duvenaud et al., 2015] worked with non-aligned graphs, $\bm{X}_{ni}\neq\bm{X}_{n^{\prime}i}$, modeling node features conditioned on its neighbors. They provided a convolutional neural network approach which is invariant to node permutation. However, unlike us, their method assumes knowledge of node features $X_{ni}$. Overall, the closest work to our model is from Mukherjee et al. [2017] where a similarity measure is used to align communities across graphs. In our approach, the model does not rely on a distance to compare graph communities as we cluster nodes across graphs jointly.

6 Synthetic experiments

In this section, we evaluate our algorithm for the joint SBM model with synthetic data where the ground truth is known. We generate data using the following generative process:

[TABLE]

For all experiments, we use $K=6$ and the same $\bm{\Theta}$, shown in Figure 1 (left). We vary hyperparameters including the number of graphs $N$, the individual graph sizes $|V_{n}|$, and the heterogeneity of clusters sizes in each graph $\alpha$. Recall that $\alpha\approx 0$ corresponds to a homogeneous setting (similar $\bm{\pi}$ across all graphs), and that as $\alpha$ increases the clusters become more heterogeneous.

We evaluate the performance of our joint spectral clustering algorithm (JointSpec), and compare against separately running spectral SBM on the individual graphs and then aligning (IsoSpec). We also include as a baseline node2vec Grover and Leskovec [2016], which embeds the nodes into a low-dimensional vector space and clusters the embeddings. For IsoSpec, we used the Blockmodels R package Leger [2016], and for node2vec we used a Python implementation Grover and Leskovec [2016].

First, we assess community retrieval performance and global $\bm{\Theta}$ estimation for each model. We design two sets of experiments, one for each assessment objective:

Community retrieval $\widehat{\bm{X}}_{n}$:

1. (a)

Fixed $\alpha\!=\!1$. $N\!=\![5,50,100,200]$. For each $N$, $|V_{n}|\!=\![25,50,100,200,500]$. Figure 1(right) shows the $\pi_{n}$s, i.e. the proportion of blocks for each graph per scenario; 2. (b)

Fixed $N\!=\!100$ and $|V_{n}|\!=\!200$. We generate datasets for varying values of $\alpha\in[0.1,2]$; 2. 2.

Global $\widehat{\bm{\Theta}}$:

1. (a)

Fixed $\alpha\!=\!1$. $N\!=\![50,200,400,600,800,1000]$. For each $N$, $|V_{n}|\!=\![25,50,200,500]$.

We measured performance both quantitatively, using Normalized Mutual Information (NMI) and Standardized Square Error (SSE), and qualitatively, by visualizing the estimated connectivity matrix for each approach. For multiple graph datasets, we measure individual graph NMIs as well as the overall NMI across all graphs, in each case, comparing the estimated membership matrices $\widehat{\bm{X}}_{n}$ with the ground truth $\bm{X}_{n}$. To measure the quality of the estimated connectivity matrix $\bm{\Theta}$, we used the standardized square error, the square error normalized by the true variance. Thus,

[TABLE]

We normalize the square error by the true variance because very high (or very low) edge probabilities have lower variances and need to be up-weighted accordingly. For good $\bm{\Theta}$ estimates, we expect Eq. (23) to be close to zero.

IsoSpec and node2vec are by construction nonaligned. In order to align them, we (1) rank each community on each graph based on $\text{diag}(\bm{\Theta}_{n})$, then (2) re-order the connectivity matrix and membership accordingly.

6.1 Results

Community retrieval $\widehat{\mathbf{X}}_{n}$, fixed $\alpha=1$:

Figure 2(left) shows the NMI curves for increasing number of nodes for different numbers of graphs (top row: individual NMIs, and bottom row: overall NMI). For individual NMIs, each data point records the median over the graphs and the shaded region shows the interquartile range. We see that JointSpec outperformed IsoSpec and node2vec for both overall and individual NMI. That the overall NMI for IsoSpecand node2vec is poor is unsurprising, given the two-stage alignment procedure involved. Interestingly however, they perform poorly on the individual NMIs as well. This indicates that regardless the choice of realignment, the use of only local information is not enough to accurately assign nodes to clusters in multiple graph domains, and that it is important to pool statistical information across graphs.

In terms of estimating the connectivity matrix, node2vec performs worst and the JointSpec estimates are the most similar to the true connectivity. Figure 4 (top row) shows the estimated connectivity matrix for each approach for the case of $200$ graphs, $500$ nodes per graph.

Community retrieval $\widehat{\mathbf{X}}_{n}$, fixed $N\!=\!100$ and $|V_{n}|\!=\!200$:

Here, we evaluate cluster retrieval performance as the cluster sizes became more heterogeneous (e.g. $\alpha$ increases). Figure 2(right) shows the NMI curves. The results suggest that IsoSpec  and JointSpec have similar performance when the graphs are balanced in terms of distribution of nodes over communities (low values of $\alpha$). However, the NMI curves diverge as $\alpha$ increases and for more heterogeneous settings (unbalanced graphs), the Joint model outperforms IsoSpec and node2vec by a large amount, both for overall and individual cluster retrieval performance.

Furthermore, we consider more complex re-alignment procedures, and two additional cluster retrieval performance measures (adjusted Rand index and misclutering rate). Results are in accordance with the ones shown in Figure 2(right) in which Joint SBMoutperforms the Isolated models. Appendix 9.9 presents those results.

We also investigated settings where the individual graph sizes varied over the dataset. To generate the data we used an overdispersed negative binomial distribution to sample graph sizes. The results are in Appendix 9.10. Overall, we found that varying $\alpha$ had more impact on cluster retrieval performance than varying the spread of graph sizes.

Global $\widehat{\mathbf{\Theta}}$:

Figure 3 shows the standardized square error (SSE) (23) for the $\bm{\Theta}$ estimates. (lower values mean better estimates of the true $\bm{\Theta}$). Again, JointSpec outperforms IsoSpec and node2vec in all scenarios. Even when each graph does not have many nodes (e.g. $|V_{n}|=25$), statistical pooling allows JointSpec to achieve good performance that is comparable with settings with larger nodes. These results also illustrate the consistency of our estimation scheme, with decreasing error as the number of samples increases.

7 Real world experiments

We consider two datasets: a dataset of Indian villages, and a Twitter dataset from the political crisis in Brazil.

Karnataka village dataset

This222available at https://goo.gl/Vw66H4 consists of a household census of $75$ villages in Karnataka, India. Each village is a graph, each person is a node, and edges represent if one person went to another person’s house or vice versa. Overall, we have $75$ graphs varying in size from $354$ to $1773$ nodes.

Figure 5 (top) shows some demographics proportions for each village: these are highly heterogeneous. Looking at Caste, for instance, we have that most villages consist of “OBC” and “Scheduled Caste”, but some have a very large proportion of people identified as “General”. Religion and Mother-tongue seem to have same behavior. Overall, we expect that Lemma 20 will not hold for this scenario, and expect our approach to be better suited than IsoSpec.

We consider a conservative setting of six communities ($K=6$) for all models. Here, ground truth is unknown and we cannot assess the performance of the models precisely. To quantitatively measure performance, we compare the demographic variables presented earlier (religion, caste and mother-tongue) with the communities assigned using each model. Table 1 shows the NMIs computed in this setting, we can see that JointSpec outperformed the baselines on both the overall NMI and the Individual NMI. We see in Figure 4 (middle row) that the connectivity matrices estimated using IsoSpec and node2vec are noisy and weakly structured, making interpretation difficult. The connectivity matrix that our method estimated presents a very strong structure, where the within-community probability is very large and across-community is very low.

Twitter

We crawled Twitter from April 6th to May 31st 2016 to collect tweets from hashtags of $7382$ users from both sides of the political crisis in Brazil. One side was for the impeachment of the former president, Dilma Rousseff, with the opposition describing the process as a coup. We constructed graphs based on the word co-occurrence matrices forming edge-links between words which were co-tweeted more than 20% of the time for a given user. Thus, networks represent users and nodes represent words. The resulting networks are somewhat homogeneous after controlling for the side the user is in. The sizes of the graphs vary from $25$ to $1039$ nodes.

We used four communities ($K=4$) in this experiment, and Figure 4 (bottom row) shows the estimated connectivity matrices. Now, on the surface, both JointSpec and IsoSpec have similar behavior, with node2vec estimating an extra cluster. Despite this superficial similarity, the words assigned to communities by the two models differ significantly. In order to assess words for each community, we split users between the two sides (pro and against government), and assign each word to its most frequent community across graphs. Table 2 shows the top three words per community per model. We colored words based on whether it is a more pro government (red), against it (blue) or neutral (black). For JointSpec, we see that community 1 has important key arguments per side such as “naovaitergolpe” (no coup) and “foradilma”(resign dilma). We also see that community 2 seems to have some stopwords such as “hora”(time) and “compartilhar” (share), and community 3 consist of aggressive and pejorative terms each side uses against the other like “elite” and “coxinha”. For IsoSpec, no such inferences are made, and words do not seem to reflect a clear pattern. node2vec had the least informative clustering performance, where the top words for the different communities were all neutral.

Since there is partial alignment across the Twitter graphs (i.e., words can appear as nodes in multiple graphs), we can assess the consistency of the clustering approaches by evaluating the entropy of community assignments across the graphs (per word). Although a word can be used in multiple contexts, we expect that a “good” clustering will assign words mostly to the same clusters, therefore resulting in a lower entropy. The results are in Appendix 9.11. Overall, we found that JointSpec had the lowest entropy.

8 Conclusion

In this work, we consider the problem of multiple graph community detection in a heterogeneous setting. We showed that we need to jointly perform stochastic block model decompositions in order to be able to estimate a reliable global structure. We compared our methods with a two-step approach we called the Isolated SBM, and node2vec. Our method outperformed the baselines on global measures (overall NMI and SSE of the connectivity matrices), but interestingly also on local measures (individual NMI). This demonstrates that our method is more accurately able to assign nodes to clusters regardless of the choice of re-alignment procedure. Overall, the Isolated SBM does not pool global information in the inference step which indicates that it can only be used in highly homogeneous scenarios.

9 Appendix

9.1 Notations

9.2 Derivation of Eq.(6) and unbiasedness

Recall that the connectivity matrix $\bm{\Theta}$ consists of edge probability for within and between communities. Now, say $\theta_{kl}$ is the element of $\bm{\Theta}$ on the $k$th row $l$-th column. Thus, one can estimate $\theta_{kl}$ by counting the number of edges between communities $k$ and $l$ and dividing by the total number of possible edges. For $k\neq l$, the total number of possible edges is the number of nodes in $k$ multiplied by the number of nodes in $l$. For a adjacency matrix $\bm{A}_{n}$, we can generalize the estimation of the connectivity matrix using matrix notation as

[TABLE]

If assume we know the true membership matrix (i.e. $\widehat{\bm{X}}=\bm{X}_{n}$), the off-diagonal elements of $\widehat{\bm{S}}_{n}$ in Eq.  (24) above have unbiased estimates, however the diagonal elements (i.e. the within community probability) are biased. More specifically, the total possible number of edges for nodes in the same communities is being assumed to have self loops which is incorrect in this setting, and also the term $\widehat{\bm{X}}_{n}^{T}\bm{A}_{n}\widehat{\bm{X}}_{n}$ is double counting the edges within communities. Formally,

[TABLE]

Here, we are using the fact that $\mathbb{E}[\bm{A}_{n}]=\bm{P}_{n}-\text{diag}\left(\bm{P}_{n}\right)$. Furthermore, we can construct an unbiased estimator for $\bm{\Theta}$ by adding the following term to each diagonal element of $\widehat{\bm{S}}_{n}$:

[TABLE]

where $|G_{nk}|$ is the number of nodes in community $k$. For all $k$, we have Eq.(26) in matrix notation as

[TABLE]

Now, it follows from Eq. (25) that

[TABLE]

Using Eqs.(24) and (27), we get the expression in Eq.(6). And using Eq. (25) and  (28), we have

[TABLE]

We also have that

[TABLE]

where $|G_{nk}|$ is the number of nodes of graph $n$ in cluster $k$.

9.3 Proof of Lemma 3.1

Proof From Equation (10), and since $\bm{X}_{n}=\bm{X}_{n*}$, we have:

[TABLE]

where $\bm{Q}_{n}^{*}:=\bm{Q}_{n}\sqrt{\frac{|V|}{|V_{n}|}}$. Recall that $\bm{Q}_{n}$ corresponds to the data from a single graph. $\bm{Q}_{n}^{*}$ is then a weighted version, based on the relative number of nodes in the graph.

From this we can transform Eq. (11) to

[TABLE]

where

[TABLE]

$\hfill\qquad\qquad\qquad\qquad\qed$

9.4 Proof of Lemma 3.2

Proof

[TABLE]

where

[TABLE]

Recall that $G_{nk}$ is the set of nodes from $n$ that are in cluster $k$, and $G_{.k}$ is the set of nodes from all graphs in cluster $k$. The last inequality (13) uses the fact that $\bm{Z}$ and $\bm{Z}_{n}$ are orthogonal matrices, we demonstrate why the following inequality holds:

[TABLE]

First, we drop what is constant on both sides of the inequality first term on both sides of the inequality.

[TABLE]

Rewriting LHS of Eq. (33) using trace operator, we have

[TABLE]

The RHS is given by

[TABLE]

Finally, the inequality in Eq. (33) holds because the difference between the RHS and the LHS is positive. Given $\bm{Z}$ is orthonormal, we have

[TABLE]

∎

9.5 Proof of Lemma 20

Proof For graph $n$, the vector of counts of nodes in each cluster has expected value given by $\mathbb{E}\left[\sum_{i}^{|V_{n}|}\bm{X}_{ni}\right]=|V_{n}|\bm{\zeta}$. Assuming the same distribution of the nodes over cluster for all graphs, $\mathbb{E}\left[\sum_{n}^{N}\sum_{i}^{|V_{n}|}\bm{X}_{ni}\right]=|V|\bm{\zeta}$. We know that $\sum_{i}^{|V_{n}|}\bm{X}_{ni}=\text{diag}(\bm{X}_{n}^{T}\bm{X}_{n})=\text{diag}(\Delta_{n}^{2})$ and $\sum_{i}^{|V|}\bm{X}_{ni}=\text{diag}(\bm{X}^{T}\bm{X})=\text{diag}(\Delta^{2})$. Defining $\alpha_{n}=|V_{n}|/|V|$ for all $n\in[1,...,N]$, we have

[TABLE]

Note that if all graphs have the same size, $|V_{n}|$, then $\alpha_{n}=N^{-1}$. Furthermore, using the eigendecomposition on both sides, we have

[TABLE]

Thus, $\bm{Z}_{n}=\bm{Z}\iff\bm{D}_{n}=\alpha_{n}\bm{D}$.

Finally,

[TABLE]

∎

9.6 Proof of Lemma 21

Proof By Lemma 20,

[TABLE]

where $\bm{Q}_{n}^{*}=\bm{Q}_{n}\frac{|V|}{|V_{n}|}=\bm{U}_{n}\bm{D}\frac{|V_{n}|}{|V|}\frac{|V|}{|V_{n}|}=\bm{U}_{n}\bm{D}$. Given $\frac{|V_{n}|}{|V|}=\frac{1}{N}$ and dropping all constants across graphs, we have $\mathbb{E}[\eta_{n}(\bm{X}_{n},\bm{W})]\propto\|\bm{X}_{n}\bm{W}_{n}-\widehat{\bm{U}}_{n}\|_{F}^{2}$           ∎

9.7 Toy data example

As an example to illustrate the effect of $\bm{Z}_{n}^{T}\Delta_{n}^{-1}\Delta\bm{Z}$ on the individual eigendecomposition, consider three graphs, where each graph is a village, nodes represent individuals and edges represent relationships between them. Assume that individuals are clustered in four different blocks based on their personalities, which reflects how they form relationships. Figure 6(left) shows the connectivity matrix based on those personalities. Also, consider that each village has its own distribution of people over the clusters, as shown, for instance, in Figure 6 (right).

Now, using the adjacency matrix eigendecomposition for each village $\bm{A}_{n}=\widehat{\bm{U}}_{n}\widehat{\bm{D}}_{n}\widehat{\bm{U}}_{n}^{T}$, we have $\widehat{\bm{Q}}_{n}=\widehat{\bm{U}}_{n}\widehat{\bm{D}}_{n}$ as a proxy for $\bm{Q}_{n}$. Since $\widehat{\bm{U}}_{n}\widehat{\bm{D}}_{n}\widehat{\bm{U}}_{n}^{T}=\widehat{\bm{T}}_{n}\widehat{\bm{D}}_{n}\widehat{\bm{T}}_{n}^{T}$ for $\bm{T}_{n}=-\bm{U}_{n}$, we consider $|\widehat{\bm{Q}}_{n}|$ instead. Figure 7 (left) shows the two largest absolute eigenvectors of each adjacency matrix. Each block has a different center of mass depending on which village (graph) it is in. This is due the fact that villages have completely different distribution of nodes over the blocks. Therefore, sharing information across villages is fundamental not only to assign underrepresented nodes to the correct block, but also to map the blocks across villages. Using our proposed transformation given in Equation (10), we obtain the results shown on Figure 7 (right). We re-scale and rotate the eigenvectors in order to have an embedding of the nodes that is closer to the global eigendecomposition. Therefore, using any clustering algorithm one can correctly recover the membership for nodes across the three villages.

9.8 Consistency

The global parameter $\bm{\Theta}$ is central in order to understand Multi-graph settings. It not only gives an overall summary of how nodes connect based on their communities, but $\bm{\Theta}$ also has the role to link all graphs. In other words, a reliable estimate of $\bm{\Theta}$ means we can predict edges between nodes in different graphs with confidence. Here, we discuss the asymptotic behavior of $\bm{\Theta}$ in Multi-graph joint SBM as $N\rightarrow\infty$. And, we show that the estimator $\widehat{\bm{\Theta}}=N^{-1}\sum_{n}\widehat{\bm{\Theta}}_{n}$ in Eq. (6) converge to the global $\bm{\Theta}$ almost surely for when we know the true membership. Lemma 9.1 (below) formalizes these statements.

Lemma 9.1**.**

Let the pair $(\bm{X},\bm{\Theta})$ parametrize a SBM with $K$ communities for $N$ graphs where $\bm{X}$ contains the membership matrix of all graphs stacked and $\bm{\Theta}$ is full rank. Write $\nu\leq\min_{n}|V_{n}|$. Now if assume $(\widehat{\bm{W}},\widehat{\bm{X}})$ is the optimal solution of Eq. 11 then $\widehat{\bm{\Theta}}$ converge to $\bm{\Theta}$ in probability Eq. (36). If we also assume $\widehat{\bm{X}}_{n}=\bm{X}_{n}$ then $\widehat{\bm{\Theta}}$ converge to $\bm{\Theta}$ almost surely Eq. (37).

[TABLE]

[TABLE]

Proof Eq.(36) follows directly from the fact that $\bm{A}_{n}$ converge in probability to $\bm{P}_{n}$ for large $|V_{n}|$, Theorem 5.2 Lei et al. [2015]. Eq.(37) follows from Eq. (29), we know that $\mathbb{E}[\widehat{\bm{\Theta}}_{n}]=\bm{\Theta}$. Thus, using Kolmogorov-Khintchine strong law of large numbers, $\widehat{\bm{\Theta}}\rightarrow\bm{\Theta}$ almost surely for large $N$.

$\hfill\qquad\qquad\qquad\qquad\qed$

Eq. (37) is only true in the Multi-graph joint case. In the Isolated setting, we need to re-align the memberships across graphs which adds an extra layer of complexity. For instance, assume the re-alignment procedure consists on (1) rank each community on each graph based on $diag(\bm{\Theta}_{n})$, then (2) re-order the connectivity matrix and membership accordingly. In this case, $Var(\widehat{\bm{\Theta}})=\sum_{n}^{N}Var(\bm{\Theta}_{n})\rightarrow\infty$, unless we assume graph size to be large, i.e. $\nu\rightarrow\infty$ where $\nu\leq\min_{n}|V_{n}|$. Nevertheless, this gives weak consistency at most. In fact, this is true for any realignment procedure whose performance is a function of graph size. Figure 1 in the Synthetic experiments shows that the Joint model estimates $\bm{\Theta}$ well even for small graph settings which is not true for Isolated models.

9.9 Additional re-alignment procedures and cluster retrieval performance measures

Here, we assess the performance of IsoSpec using more complex re-alignment procedures (Iso2 and Iso3), we compare them with the one we used in the main body of the paper (Iso1). We also include adjusted rand index (ARI) [Rand, 1971] and misclutering rate (MCR) measures which are often used to evaluate cluster performance when ground truth is known. Now, we describe the re-alignment procedures:

• •

Iso1 (used in the main body of the paper)

Rank diag$(\theta_{n})$ 2. 2.

Re-order $X_{n}$ and $\theta_{n}$ accordingly.

• •

Iso2

Cluster the centers $W_{n}$ across graphs 2. 2.

Re-assign nodes to clusters based on the center of the centers.

• •

Iso3

Search over all permutation to make all $W_{n}$ closest possible from each other 2. 2.

Re-order $X_{n}$ and $\theta_{n}$ accordingly.

We compute ARI, MCR and NMI for different re-alignment procedure for increasing heterogeneity the same synthetic data presented in 2(right). In order to make these additional experiments comparable with 2(right), we also include results for Joint SBM.

Figure 8 shows that ARI and MCR have similar results of NMI for the Joint and Iso1. In terms of the additional re-alignment procedures, Iso2 seems to outperform all other methods for the Individual case if the focus is on ARI. However, it does not have a good performance using other measures, and the reason is due to the fact that Iso2 tends to accumulate nodes in lower number of clusters, i.e. $<<K$. In terms of the Overall curves, the Joint SBMoutperforms all the baselines for heterogeneous settings as it was expected.

9.10 Synthetic data experiment for varying graphs sizes

Here, we aim to assess cluster retrieval performance in settings where the graphs have different sizes. We used $|V_{n}|\stackrel{{\scriptstyle iid}}{{\sim}}NB(\mu,r)$ to sample the size of each graph, where $\mu$ is the mean and $r$ the dispersion parameter. We fixed $\mu=200$ and we vary $r\in[1,10]$. Lower values of $r$ mean more variability in graph size distribution. We also consider two main scenario: 1) homogeneous $\alpha=\frac{1}{K}$; and 2) heterogeneous $\alpha=1$. Figure 9 shows the curves for each model in each scenario. For the heterogeneous scenario it is clear that the JointSpec outperform the baselines. Also, NMI curves look flat for increasing $r$ on both scenarios (homogeneous and heterogeneous) which suggests that the distribution of nodes over clusters (controlled by $\alpha$) is more detrimental for cluster retrieval than the size of the graphs.

9.11 Assessment of community assignments in Twitter

We also compute the entropy of the community assignment per words across graphs. We expect the community of the words to be consistent across graphs, therefore a lower entropy. We found that the Joint model had the lowest entropy overall, Figure 10 shows the results. Also, we include a pairwise distribution of the difference of the entropies which shows that JointSpec had the lowest entropy for the majority of the words.