Regularized Stochastic Block Model for robust community detection in complex networks

TL;DR

This paper introduces a regularized stochastic block model that stabilizes community detection in complex networks by controlling internal degree ratios, enabling reliable inference of assortative or disassortative structures.

Contribution

A novel regularized model that guides inference algorithms to accurately detect different community structures, overcoming local optima issues in traditional models.

Findings

The regularized model converges quickly to the desired community structure.

It reliably detects assortative or disassortative patterns based on a single parameter.

It outperforms degree-corrected stochastic block models in complex networks.

Abstract

The stochastic block model is able to generate different network partitions, ranging from traditional assortative communities to disassortative structures. Since the degree-corrected stochastic block model does not specify which mixing pattern is desired, the inference algorithms, which discover the most likely partition of the networks nodes, are likely to get trapped in the local optima of the log-likelihood. Here we introduce a new model constraining nodes' internal degrees ratios in the objective function to stabilize the inference of block models from the observed network data. Given the regularized model, the inference algorithms, such as Markov chain Monte Carlo, reliably finds assortative or disassortive structure as directed by the value of a single parameter. We show experimentally that the inference of our proposed model quickly converges to the desired assortative or disassortative partition while the inference of degree-corrected stochastic block model gets often trapped at the inferior local optimal partitions when the traditional assortative community structure is not strong in the observed networks.

Tables

Table 1. Table 1: Real networks for experiments. The number of blocks for each network is set to its value generally accepted in the previous publications.

Network	#Nodes	#Edges	#Blocks	Ref.
Karate club	34	78	2	[13]
Dolphin social network	62	159	2	[14]
Characters from Les Miserables	77	254	6	[2]

Table 2. Table 2: The log-likelihoods of the three partitions of the twin stars network under the standard stochastic block model (SSBM), the degree-corrected stochastic block model (DCSBM) and our block model with regularization (RSBM). The partitions are shown in Figure  7 and the corresponding maximum log-likelihood values are marked by the bold font.

Model	Core-periphery	Twisted core-periphery	Assortative Community
SSBM	-12.47664	-18.38972	-24.66870
DCSBM	-44.66540	-39.55004	-45.82902
RSBM ($\alpha =0.3$)	-20385.51476	-377.52624	-44.68795
RSBM ($\alpha =0.6$)	-20394.46861	-368.57239	-43.38465
RSBM ($\alpha =0.9$)	-20416.64932	-346.39168	-43.35718

Equations

$$\log P(\mathbf{A}|\{{\lambda }_{ij}\})=\prod _i\frac{(\frac{1}{2}{\lambda }_{ii}{)}^{A_{ii}/2}}{(\frac{1}{2}{A}_{ii})!}{e}^{-{\lambda }_{ii}/2}\prod _{i<j}\frac{{\lambda }_{ij}^{A_{ij}}}{A_{ij}!}{e}^{-{\lambda }_{ij}}.$$

$$\log P({\bf A}|\{\lambda_{ij}\})=\frac{1}{2}\sum_{ij}\Big{(}A_{ij}\log\lambda_{ij}-\lambda_{ij}\Big{)}.$$

$${\omega }_{rs}=\{\begin{array}{ll}& \gamma {\omega }_0\text{ if }r=s,\\ & {\omega }_0\text{if }r\ne s.\end{array}$$

$${\lambda }_{ij}=\{\begin{array}{ll}{\omega }_{g_i,g_j}{I}_i{I}_j& \text{mbox}if{g}_i=g_j\\ {\omega }_{g_i,g_j}{O}_i{O}_j& \text{mbox}otherwise\end{array}$$

$$\displaystyle\mathcal{L}(G|{\bf g,\omega,I,O})=2\sum_{i}\Big{(}k_{i}^{+}\log I_{i}+k_{i}^{-}\log O_{i}\Big{)}+\sum_{rs}m_{rs}\log\omega_{rs}-\omega_{rs}\Lambda_{rs}$$

$${\Lambda }_{rs}=\{\begin{array}{ll}({\sum }_{i\in r}{I}_i{)}^2& \text{mbox}ifr=s\\ {\sum }_{i\in r}{O}_i{\sum }_{i\in s}{O}_i& \text{mbox}ifr\ne s\end{array}$$

$${\hat{\omega }}_{rs}=\frac{m_{rs}}{{\Lambda }_{rs}}$$

$$\mathcal{L}(A|\mathbf{g},\mathbf{I},\mathbf{O})$$

$$\mathcal{L}(G|\mathbf{g},\mathbf{I},\mathbf{O})$$

$$f(k)=\alpha +\frac{(1-\alpha )}{k}$$

$$f(k)=max(f,\frac{1}{k})$$

$$\text{coverage(g)}=\frac{|\{(i,j)\in E|g_i=g_j\}|}{|E|}$$

$$\sum _j{\lambda }_{ij}=k_i$$

$$\mathcal{L}={\mathbb{D}}_{KL}(p_{\text{degree}}(r,s)||p_{\text{null}}(r,s))-2{\mathbb{E}}_{k_i}[H(\frac{k_i^{+}}{k_i})]$$

$$\mathcal{L}(G|\mathbf{g})=\sum _{rs}{m}_{rs}\log \frac{m_{rs}}{{\kappa }_r{\kappa }_s},$$

$$\mathcal{L}(G|\mathbf{g})=\sum _{rs}\frac{m_{rs}}{2m}\log \frac{m_{rs}/2m}{({\kappa }_r/2m)({\kappa }_s/2m)},$$

$$p(r\to s|t)=\frac{m_{ts}+ϵ}{{\sum }_s{m}_{ts}+ϵB}$$

$$a=\min\Bigl{\{}\exp(\Delta\mathcal{L})\frac{\sum_{t}n_{t}p(s\to r|t)}{\sum_{t}n_{t}p(r\to s|t)}\Bigr{\}},$$

$$\mathcal{L}(G|\mathbf{g},\mathbf{I},\mathbf{O})$$

$$\frac{k_i}{{\hat{\theta }}_i}=\frac{m_{rr}}{{\sum }_{i\in V_r}{f}_i{\hat{\theta }}_i}{f}_i+\frac{{\sum }_{s:s\ne r}{m}_{rs}}{{\sum }_{i\in V_r}(1-f_i){\hat{\theta }}_i}(1-f_i)$$

$$\sum _j{\lambda }_{ij}$$

$$=I_i\frac{m_{g_i,g_i}}{{\sum }_{j\in g_i}{I}_j}+O_i\frac{{\sum }_{b:b\ne g_i}{m}_{g_i,b}}{{\sum }_{j\in g_i}{O}_j}$$

$$\frac{\partial \mathcal{L}}{\partial I_i}$$

$$\frac{\partial \mathcal{L}}{\partial O_i}$$

$$\forall i\in r:\frac{k_i^{+}}{I_i}\sum _{i\in r}{I}_i+\frac{k_i^{-}}{O_i}\sum _{i\in r}{O}_i=\sum _{i\in r}{I}_i+O_i={\kappa }_r$$

$$\mathcal{L}=\sum _{rs}\frac{m_{rs}}{2m}\log \frac{m_{rs}/2m}{{\Lambda }_{rs}/(2m{)}^2}-2\sum _i\frac{k_i}{2m}(H(\frac{k_i^{+}}{k_i}))$$

$$p_{\text{null}}(g_i=r,g_j=s)=\{\begin{array}{ll}& \frac{{\kappa }_r^{-}}{2m}\frac{{\kappa }_s^{-}}{2m}\text{if}r\ne s\\ & \frac{{\kappa }_r^{+}}{2m}\frac{{\kappa }_s^{+}}{2m}\text{if}r=s\end{array}$$

Full text

Regularized Stochastic Block Model for robust community detection in complex networks

Xiaoyan Lu1, Boleslaw K. Szymanski1,2111E-mail: szymab@rpi.edu

1 Social and Cognitive Networks Academic Research Center and Department of Computer Science, Rensselaer Polytechnic Institute, Troy NY 12180, USA

2 Społeczna Akademia Nauk, Łódź, Poland

Abstract

The stochastic block model is able to generate different network partitions, ranging from traditional assortative communities to disassortative structures. Since the degree-corrected stochastic block model does not specify which mixing pattern is desired, the inference algorithms, which discover the most likely partition of the networks nodes, are likely to get trapped in the local optima of the log-likelihood. Here we introduce a new model constraining nodes’ internal degrees ratios in the objective function to stabilize the inference of block models from the observed network data. Given the regularized model, the inference algorithms, such as Markov chain Monte Carlo, reliably finds assortative or disassortive structure as directed by the value of a single parameter. We show experimentally that the inference of our proposed model quickly converges to the desired assortative or disassortative partition while the inference of degree-corrected stochastic block model gets often trapped at the inferior local optimal partitions when the traditional assortative community structure is not strong in the observed networks.

Keywords: community detection, stochastic block model, Markov chain Monte Carlo, modularity, assortative structures

Introduction

The study of modular structure in networks has a long history in the literature [4, 2, 3, 1]. The primary focus of this line of work has been the discovery of the assortative community structures where the edges inside each module are generally more numerous than the edges across them. In complex networks, the modular structures in networks include not only such assortative community structures but also other mixing patterns, like the core-periphery structures [5] and bi-partite structures. These various structures found in networks demand a more comprehensive class of community detection models than the assortative ones, targeted by the modularity-based approaches [2]. Modularity maximization is perhaps the most widely used approach for community detection. It aims at maximizating the modularity of the network partitions, which is a broadly accepted quality metric comparing the number of edges observed in each community to the expected number in a random graph with the same degree sequence. As shown in [6], modularity maximization is, in fact, equivalent to the maximum likelihood estimation of the degree-corrected planted partition model, a special case of the degree-corrected stochastic block model [9]. Therefore, the stochastic block model can be considered more general than the traditional modularity-based community detection models because it can generate a wide variety of different network structures, which include not only the traditional assortative community structures, but also the disassortative structures such as the core-periphery structures [5]. Exceeding the limitations of modularity maximization approaches, the statistical inference of stochastic block model, which maximizes the likelihood of generating the observed network data, is capable of discovering of both assortative and disassortative structures in networks. The stochastic block model and its variants have gained significant attention in the past decade.

In the following, when we discuss the nodes’ block assignment, for any node $l$, its block assignment is denoted by $g_{l}$. The standard stochastic block model is a variant of the generative random graph model which assumes the probability of connecting two nodes in a graph is defined exclusively by their block assignments. More precisely, this probability follows Bernoulli distribution with the mean $\omega_{g_{i},g_{j}}$. Hence, the model is fully specified by the block assignments of nodes $\{g_{i}\}$ and the mixing matrix $\Omega=\{\omega_{rs}\}$ governing the probabilities of observing one edge between each pairs of nodes from blocks $r$ and $s$. If the diagonal elements $\omega_{rr}$ in the mixing matrix are larger than the off-diagonal elements, then the networks generated by the stochastic block model have the traditional assortative communities. When the off-diagonal elements $\omega_{rs}$ for $r\neq s$ are larger than the diagonal elements, the generated network contains disassortative mixing patterns, such as the structures observed in the core-periphery graphs [5] and the bi-partite graphs. In general, the inference of the stochastic block model aims at discovering the mixing matrix and nodes block assignment which maximize the likelihood of generating the observed network. It does not impose any constraint on which type of structure is discovered. Thus, the inference of the stochastic block model is more general than the traditional modularity maximization because it may recover a set of assortative or disassortative structures from the observed network.

Given the nodes’ block assignments of the standard stochastic block model, the nodes in the same block are by definition statistically indistinguishable, and they all have the same expected node degrees. Therefore, the most likely block assignments often cluster the nodes of similar degrees into a block, resulting in clusters with homogeneous node degrees, rather than heterogeneous node degrees in each block, typical for the traditional community structures. The degree-corrected stochastic block model [9] defines the expected number of edges between nodes $i$ and $j$ as $\lambda_{ij}=\omega_{g_{i},g_{j}}\beta_{i}\beta_{j}$ where for each node $l$, $\beta_{l}$ is a model parameter associated with it. Note that $\lambda_{ij}$ is defined as the expected number of edges and not as customary the probability, assuming multi-edges (multiple edges between a pair of nodes) are allowed in this definition [9]. The expected node degrees in the degree-corrected stochastic block model, given the maximum likelihood estimates of these model parameters, are equal to the degrees observed in the input network. Hence, the degree-corrected stochastic block model allows a community to have a wide range of node degree distribution.

The simple yet effective degree-correction modification improves the performance of statistical inference of community structure in complex networks. When the nodes in an assortative community have a wide range of degree distribution, the statistical inference of the standard stochastic block model is likely to detect the sub-optimal disassortative structures, while the statistical inference of the degree-corrected model is more likely to discover the sought-after assortative structures. Yet, the degree-corrected stochastic block model does not impose any constraints on the mixing pattern of the resulting block assignments, thus the return of the assortative structures is not guaranteed.

As suggested in [10], both the traditional assortative community structures and the other disassortative structures are potentially good fits to the degree-corrected stochastic block models. Hence, depending on the starting point, the inference algorithm like Markov chain Monte Carlo tends to get trapped in the local optimum of the log-likelihood. Therefore, in this paper, we propose a regularized stochastic block model which provides an extra regularization term to guide the discovery of assortative or disassortative structure by the statistical inference of the stochastic block model. Unlike the modularity maximization algorithm which always attempts to find traditional assortative communities, the inference of this regularized stochastic block model controls the mixing patterns discovered in the given network.

Results

Degree-corrected stochastic block model

The degree-corrected stochastic block model [9] is a generative model of the graph in which the edges are randomly placed between nodes. The probability of observing an edge between two nodes depends exclusively on the block assignments of its two endpoints. In the degree-corrected stochastic block model, the nodes in the same block are statistically indistinguishable from each other. The degree-corrected stochastic block model assumes the number of edges between nodes $i$ and $j$ is independently distributed Poisson random variable with the mean $\lambda_{g_{i},g_{j}}$.

Let $\bf A$ be the adjacency matrix whose component $A_{ij}$ denotes the number of edges between nodes $i$ and $j$ in an unweighted undirected multigraph. Multi-edges (multiple edges between a par of nodes) and self-loop edges (the edges connecting a node to itself) are practical in certain networks such as the web network where a web page may contain multiple hyperlinks pointing to other pages and to itself. Such edges are less common in social networks. However, most social networks are very sparse, so the impact of multi-edges and self-loop edges is negligible in such networks. Let a node $i$ with $k$ self-loop edges be represented by the diagonal adjacency matrix element $A_{ii}=2k$ and the number of edges between two different nodes $i$ and $j$ follows the Poisson distribution with mean $\lambda_{ij}$ while the number of self-loop edges at node $l$ follows the Poisson distribution with mean $\frac{1}{2}\lambda_{ll}$.

Given the parameters $\{\lambda_{ij}\}$, the likelihood of generating $\bf A$ is

[TABLE]

Note that $\lambda_{ij}$ is defined as the number of edges here and not as customary the probability, because multi-edges are allowed in this definition. In an unweighted undirect network, after ignoring all terms independent of the $\lambda_{ij}$s, the log-likelihood above simplifies to

[TABLE]

In the degree-corrected stochastic block model, the model parameter $\lambda_{ij}$ is defined as $\lambda_{ij}=\omega_{g_{i},g_{j}}\beta_{i}\beta_{j}$ where for each node $l$, $\beta_{l}$ is a parameter associated with this node. Given a partition of the network, i.e., its block assignments $\{g_{i}\}$, reference [9] obtains the maximum likelihood estimates of the model parameters as $\hat{\beta}_{i}=\frac{k_{i}}{\kappa_{g_{i}}}$ and $\hat{\omega}_{rs}=m_{rs}$, where $k_{i}$ is the degree of node $i$, $\kappa_{r}$ is the sum of the degrees of all nodes in a block $r$, and $m_{rs}$ is the total number of edges between blocks $r$ and $s$, or, if $r=s$, twice the number of edges in $r$. Note that each parameter $\omega_{rs}$ in the degree-corrected stochastic block model is not the probability of observing an edge between endpoints in blocks $r$ and $s$. Instead, it is a model parameters that depends exclusively on the block assignments $r$ and $s$.

As Eq. 2 indicates, the degree-corrected stochastic block model does not specify the mixing pattern of the resulting block assignments. It is possible that the Poisson means $\lambda_{ij}$ for the pairs of nodes in the same block are much smaller than the corresponding value between different blocks, which corresponds to the disassortative mixing pattern. Hence, the disassortative mixing pattern can also be a good fit to the degree-corrected stochastic block model. This observation inspires us to propose a regularization term for the degree-corrected stochastic block model, so it controls the mixing patterns discovered in the given network.

Assortative and disassortative structures

As defined in previous literatures [2, 3, 1], the assortative structures correspond to the traditional community structures, nodes are more frequently connected with each other inside communities than with the rest of the network. The disasortative structures do not satisfy this condition. For example, the core-periphery structures [5] divides the networks nodes into the core part, where nodes are often the hubs of the networks, and the periphery part, where nodes of low-degree connect to the core nodes. This is an advantage of the degree-corrected stochastic block model because it allows the discovery of the structures with different mixing patterns. However, when searching for the weak assortative community structures, the inference algorithm may still return the disassortative structure[11] instead.

We generate synthetic networks using the degree-corrected stochastic block model with a block assignment $\{g_{i}\}$ and the parameters $\omega_{rs}$ chosen for the assortative communities as follows

[TABLE]

where a large value of $\gamma$ results in strong community structure while $\omega_{0}$ controls the sparsity of the network. The degree sequence is drawn from a power-law distribution with exponent $2.5$, and the block assignments are randomly assigned, but each block is of the same size.

Given the synthetic networks produced by the degree-corrected stochastic block model, we infer the block assignment $\{g_{i}\}$, recovering the parameters used for its generation. Specifically, the number of communities is set as two for both the generation and recovery. To generate samples, the degree-corrected stochastic block model uses $\omega_{0}=0.01$ and $\gamma=10$. Each of the two blocks contains $10$ nodes. We scatter the sampled partitions on the x-y plane in Figure 1 and the z-axis indicates the log-likelihood of the corresponding partitions. We adapt the Markov chain Monte Carlo (MCMC) algorithm [16, 12] to infer the block assignment of the stochastic block model and its extensions. The MCMC algorithm [12] samples the distribution of all possible block assignments in the network partitions space. For each trail, the algorithm starts from a random partition and iteratively, with certain probability, traverse to an adjacent partition. An entire MCMC sweep of all nodes in the network requires $O(E)$ operations where $E$ is the number of edges in the networks [12]. The mixing time of the MCMC algorithm depends on the specific input network and the starting point. The details of the MCMC algorithm are discussed in the Supplementary Information.

Figure 1 shows the existence of multiple local optima in the log-likelihood of degree-corrected stochastic block model. The inference process finds three local optima here: (i) partition A which corresponds to the assortative structure matching the ground truth block assignment used for its generation; (ii) partition B which corresponds to a disassortative structure; and (iii) another disassortative partition which is not explicitly marked in the Figure 1(a). Under the degree-corrected stochastic block model, the MCMC inference starting from random initial partition finds the most suitable partition A in only 2 out of 20 trials, whereas the other attempts are trapped at the local optima as shown in Figure 1(b).

Since there are multiple local optima in the log-likelihood of the degree-corrected stochastic block model, the inference algorithm may converge to any of them nondeterministically. Specifically, the type of the discovered structure depends on the trial starting point and inference algorithm parameters. To avoid such nondeterministic outcomes, we introduce a novel approach called Regularized Stochastic Block Model (RSBM) applicable to any inference algorithm. RSBM constrains nodes’ internal degree ratios, each of which is defined in the objective function as the fraction of a node’s neighbors inside the same community. The resulting algorithm reliably finds assortative or disassortive structures as directed by the value of a single parameter.

Regularized stochastic block model

We extend the formulation of the expected number of edges between nodes $i$ and $j$, determined by the Poisson rate $\lambda_{ij}$, in the degree-corrected stochastic block model by defining it as

[TABLE]

where any node $l$ has two associated parameters $I_{l}$ and $O_{l}$. Given Eq. 2, the log-likelihood of generating graph $G$ by this regularized stochastic block model can be written as

[TABLE]

where $k_{i}^{+}$ is the number of neighbors of node $i$ which are inside the same block given the block assignment $\bf g$ and $k_{i}^{-}=k_{i}-k_{i}^{+}$. Thus, we get

[TABLE]

To simplify, we write $i\in r$ if $g_{i}=r$. For block assignment $\bf g$, the maximum-likelihood values of $\omega_{rs}$ are

[TABLE]

Dropping the constants and substituting using Eq. 5, we obtain

[TABLE]

Note that if we set $I_{i}=O_{i}=1$ here, the log-likelihood above reduces to the definition of standard stochastic block model with $\Lambda_{rs}=n_{r}n_{s}$ which is exactly the product of the sizes of two blocks $r$ and $s$. When $I_{i}=O_{i}=k_{i}$, the second sum on the right hand side (RHS) becomes irrelevant to the maximum likelihood estimation (MLE) result. Hence, the log-likelihood reduces to the definition of degree-corrected stochastic block model in Eq. 8 with $\Lambda_{rs}=\kappa_{r}\kappa_{s}$, i.e., the product of the sums of degrees of nodes in two blocks $r$ and $s$. Hence, by introducing two sets of parameters ${\bf I}=\{I_{i}\}$ and ${\bf O}=\{O_{i}\}$ in the edge probability, we obtain a more generalized definition of stochastic block model here.

Regularization by prior in-degree ratios

For alternative formulation of our model, we define the prior in-degree ratio $f_{i}=I_{i}/(I_{i}+O_{i})$ and $\theta_{i}=I_{i}+O_{i}$ for each node $i$. By rewriting the second summation on the RHS of Eq. 8, we get

[TABLE]

where $H(\frac{k_{i}^{+}}{k_{i}},f_{i})=-\frac{k_{i}^{+}}{k_{i}}logf_{i}-\frac{k_{i}^{-}}{k_{i}}log(1-f_{i})$ represents the cross entropy between the observed and prior in-degree ratio. Therefore, the prior in-degree ratios $\{f_{i}\}$ regularizes the in-degree ratios $\{\frac{k_{i}^{+}}{k_{i}}\}$ in the resulting partition by maximizing Eq. 9.

In real networks, the low degree nodes are more likely to have neighbors inside a block than the high degree nodes are. Suppose $f_{i}$ depends only on the degree of node $i$, i.e. $f_{i}=f(k_{i})$. Then, the function $f(k):\mathbb{Z}_{+}\to[0,1]$ should be strictly decreasing. In an assortative partition of the network, we have

• •

$f(1)=1$ because a node with degree one must connect to the community it belongs to;

• •

for $k\approx|V|$, $f(k)\ll 1$ because a super-hub eventually does not belong to any community as its degree is of the order of the number of nodes in the entire network.

A simple function $\{f(k)\}$ satisfying this requirement is of the form

[TABLE]

where $\alpha$ is the only extra parameter we introduce to the regularized stochastic block model (RSBM). Alternatively, we can select a constant $f\in(0,1)$ such that

[TABLE]

The impact of different choices of $f_{i}$ on the discovered block assignment is discussed in the following two subsections presenting experimental results. It is worth noting that, for either choice of the prior in-degree ratio $\{f_{i}\}$, there is only one extra parameter, i.e. $\alpha$ or $f$, introduced here in addition to the original parameters of the degree-corrected stochastic block model.

Experimental Results

We generate synthetic networks with the assortative communities using the degree-corrected stochastic block model. The parameters $\omega_{rs}$ in the network generation process are specified by Eq. 3. By selecting a small value of $\gamma>1$ in Eq. 3, the generated community structures are relatively weak, thus, it is more difficult to detect them using the statistical inference of the degree-corrected stochastic block model.

We adapt the Markov chain Monte Carlo (MCMC) algorithm [16] to infer the block assignment of the stochastic block model and its extensions. Figure 2(a) shows there is only one unique local optimum, the partition C, found by 20 MCMC trials under the regularized stochastic block model. Therefore, all 20 MCMC trials converge to this unique local optimum. As shown in Figure 2(b), the Markov Chain Monte Carlo inference finds the correct block assignment within only 150 steps. The regularization terms made it possible to avoid the unsuitable local optima during the inference.

We use the networks including Karate club network of Zachary [13], the Dolphin social network of Lusseau et al. [14] and the network of fictional characters’ interactions in the novel Les Miserables by Victor Hugo [2] to demonstrate the performance of the regularized model introduced here. The details of each network are presented in the Supplementary Materials. For Karate club network, we evaluate the effect of the regularization terms on the resulting partitions using Markov chain Monte Carlo (MCMC) as the inference algorithm. For every node $i$ in the network, we set the parameter $\theta_{i}=k_{i}$ and $f_{i}=max(f,1/k_{i})$ for our regularized stochastic block model defined by Eq. 9 where $k_{i}$ is the degree of node $i$ and $f_{i}$ represents the prior in-degree ratio for regularization. Figure 3 shows the most likely partition of the Karate club network found by MCMC using different $f$ values. The color represents the block assignment, and the black dashed line divides the network into two parts in the ground truth partition. As shown in Figure 3, when $f=0.14$, the inference algorithm outputs a core-periphery structure which clusters high-degree nodes into the blue block and the remaining low-degree nodes into the red block. This is because the sum of cross entropy terms serves as a regularization term which penalizes those partitions that assign adjacent nodes into the same block. As the value of $f$ grows, the inference algorithm is more likely to detect assortative structure. When $f=0.85$, the inferred block assignment matches the ground truth partition of the Karate club network with the exception of one single red node. However, this node has only one connection to each block; thus, it is quite arguable to which block this node should belong.

The results in Figure 4 show that the MCMC inference of our RSBM model on these networks finds partitions with higher coverage than the ones inferred by the degree-corrected stochastic block model. The coverage of a partition [1] is defined as the ratio of the number of edges with both endpoints in the same block to the total number of edges in the entire network

[TABLE]

A low coverage indicates that the resulting partition is disassortative. An ideal assortative partition of the network, where all clusters are disconnected, yields a coverage of 1.

Figure 4 shows the coverages of the partitions found in the three real networks mentioned above. We randomly remove edges in these networks to further increase the sparsity of these networks. And the numbers of blocks used in trials are set to their values for these real networks well-accepted in the literature. The results in Figure 4 indicate that, under degree-corrected stochastic block model (DCSBM), the inference algorithm is likely to miss the assortative structures and returns instead the disassortative partitions of the network, which also fit the model in such cases. In contrast, the MCMC inference of our regularized stochastic block model (RSBM) almost deterministically produces the assortative structures. Interestingly, the inference of degree-corrected stochastic block model produces the partitions with the coverage values distributed at two levels. There is no partition with a coverage between these two levels found by the inference algorithm. This is similar to the case of the synthetic network in Figure 1 where both the assortative communities and disassortative structures fit the degree-corrected stochastic block model. Which structure is found is determined by random sampling of the potential partitions. In other words, there is no way to guide whether the disassortative structures or the assortative communities are preferred, so, two consecutive MCMC trials may return completely different structures.

In contrast, the inference of the introduced here regularized stochastic block model using the prior in-degree ratio $f_{i}=0.8+0.2/k_{i}$ is very robust. It only produces partitions with a high coverage, which are generally distributed at the same level as the assortative partitions under degree-corrected stochastic block model. Moreover, the sparsity of the networks does not have an obvious impact on the resulting partitions under our regularized model.

We evaluate the modularity [15] of resulting partitions of the real networks as a function of parameter $f$. A high modularity indicates the strong community structure. In our experiments, we use a constant $f$ such that $f_{i}=f$ for each node $i$ in the network with degree $k_{i}>1/f$, otherwise $f_{i}=1/k_{i}$. We start with a small $f$ value and increase it in each iteration. The MCMC inference uses as a starting point the original network initially, and the network partition found in the previous iteration subsequently. Figure 5c shows that, as the value of $f$ increases, in general both the modularity and the coverage grow. When $f$ is close to 0, the value of $f$ does not have any effect on the network partition. However, as $f$ becomes larger, then at a certain threshold of about 0.75, the increase of $f$ leads to a critical transition from disassortative partitions to assortative communities. Then, a larger $f$ value does not further increase the modularity of the partition. These results indicate that, with one single parameter $f$, the Markov Chain Monte Carlo inference gains the flexibility to choose between assortative communities resulting from clustering dense modules of nodes together or disassortative structure, such as the core-periphery structure from clustering high-degree and low-degree nodes into separate blocks. With a choice of high $f$, the inference algorithm is likely to detect assortative communities.

Discussion

The stochastic block model is able to produce a wide variety of different network structures, including traditional assortative communities and different from them disassortative structures. In theory, it should be possible to guide the inference algorithm which type of structures is preferred when both types of structures fit the stochastic block model and its degree-corrected variant well for the input network. However, the existence of multiple local optima of the log-likelihood traps the inference algorithms in one of them. Although there were efforts to enable community detection at different levels of granularity, cf. [7, 21], the need for controlling assertiveness of the solution has not been addressed so far. Here, we apply a simple yet effective constraint on nodes’ internal degree ratio in the objective function. This approach is independent of the inference algorithm. The resulting algorithm reliably finds assortative or disassortive structure as directed by the value of a single parameter $f$. We validated the model experimentally testing its performance on several real and synthetic networks. The experiments show that the inference of degree-corrected stochastic block model quickly converges to the sought-after assortative or disassortative structure. In contrast, the inference of degree-corrected stochastic block model gets often trapped in the inferior local optimal partitions.

Methods

Proving properties of the Regularized Stochastic Block Model

Theorem 1**.**

When $f_{i}=1/2$ for each node $i$ in the network, the MLE of our RSBM model defined by Eq. 9 becomes the MLE of degree-corrected stochastic block model222The proofs of Theorem 1,2 and 3 are provided in the Supplementary Material..

According to Theorem 1, if we add a constraint $I_{i}=O_{i}$ for every $i$, the model introduced here reduces to the degree-corrected stochastic block. Figure 6 illustrates the relationship between the most commonly used variants of stochastic block models. If we plot the constraints of these models on a 2D plane with $I_{i}$ and $O_{i}$ as the $x$ and $y$ axes, respectively, the standard stochastic block model (SSBM) sets $I_{i}=O_{i}=1$ for each $i$, which maps to the point (1,1) in the 2D coordinate plane. The constraint $I_{i}=O_{i}$ which represents the degree-corrected stochastic block model (DCSBM) maps onto the line with a slope $f_{i}=1/2$ in the 2D coordinate plane. Here, we extend the stochastic block model in two different directions: (i) using a constraint on $\theta_{i}=I_{i}+O_{i}$; (ii) choosing $I_{i}$ and $O_{i}$ on the line with a customized slope $f_{i}$. It turns out the second choice, like degree-corrected stochastic block model, preserves the degree sequence of the network.

Theorem 2**.**

Given any customized $\{f_{i}\}$ and the corresponding maximum-likelihood estimator $\hat{\theta}_{i}$, our model defined by Eq. 9 preserves the degree sequence of a network, so the expected degree of node $i$ generated by the RSBM model with $\hat{\theta}_{i}$ is

[TABLE]

Information-theoretic interpretations

While the maximum likelihood estimator $\hat{\theta}_{i}$ preserves the degree sequence in the observed network, we find that the closed-form analytic expression of such estimator is hard to obtain. However, when we impose constraints that for each $i$, $\theta_{i}=k_{i}$ and look for the MLE of $f_{i}$ for every $i$, the log-likelihood of the model introduced here has an interesting interpretation uncovered by the following theorem.

Theorem 3**.**

When $\theta_{i}=k_{i}$ for every node $i$, maximizing the log-likelihood of Eq. 9 is equivalent to maximizing the target function

[TABLE]

where the first term represents the Kullback–Leibler divergence between the edge distribution under the block model and the corresponding distribution under the null model which randomizes the edges inside and across blocks respectively, while the second term is twice the expectation of the binary entropy of in-degree ratio $k_{i}^{+}/k_{i}$.

Imposing the constraint $I_{i}+O_{i}=k_{i}$, we obtain a new model which involves a regularization term of the expected entropy of in-degree ratio $k_{i}^{+}/k_{i}$. According to Theorem 3, maximizing the log-likelihood of this model has a physical interpretation: the MLE of the RSBM model increases the KL divergence between the edge distribution under the block model and the corresponding edge distribution under the null model, and, at the same time, it decreases the expected entropy of the in-degree ratio $k_{i}^{+}/k_{i}$. Intuitively, the null model here separates the edges inside a block and the edges across different blocks. It assumes all the edges inside blocks are statistically equivalent, and so are all the edges across different blocks. In contrast, the null model of the degree-corrected stochastic block model [9] mixes all the edges, assuming all the edges are statistically equivalent - the KL divergence term in its null model does not distinguish the edges inside the same block and those across different blocks. On the other hand, the sum of entropy function $H(k_{i}^{+}/k_{i})$ controls the identification of the edges’ types. In Eq. 14, $H(k_{i}^{+}/k_{i})$ is the entropy function which achieves its minimum when $k_{i}^{+}/k_{i}\to 0$ or $k_{i}^{+}/k_{i}\to 1$. Therefore, the MLE of this model is likely to classify the edges of a node as either all in-edges or all out-edges – the former case is likely to detect assortative communities and the latter case infers disassortative structure as the most probable block assignment.

For community detection problems, it is rarely observed that all neighbors of a node are not located in the same block, especially for those nodes with low degrees. Unlike traditional community detection algorithms, the definition of stochastic block model and its variants mentioned above does not explicitly control the portion of neighbors in different blocks, rather they rely on the specific statistical inference to determine the block assignments. This observation inspires us to regularize the traditional stochastic block model on the in-degree ratio. Specifically, our regularized stochastic block model (RSBM) introduced here maximizes the objective function of Eq. 9 with $\theta_{i}=k_{i}$ and $f_{i}$ defined by the prior in-degree ratio. As explained above, the KL divergence of RSBM has similar meaning as that of degree-corrected stochastic block model. The expected cross-entropy term serves as a regularization term to control the resulting partition. Intuitively, when $f_{i}$ is close to one, then the inference algorithm tends to cluster nodes in a network into dense modules; when $f_{i}$ is close to zero, then disassortative partitions such as the core-periphery structure are likely to be discovered because the regularized block model tends to assign adjacent nodes into different blocks to decrease $k_{i}^{+}$.

Data availability statement

The network data sets used for the evaluation of our proposed model are available at http://konect.uni-koblenz.de/networks/

Acknowledgment

The authors express thanks to Dr. Santo Fortunato for discussions of degree-correlated SBM. This work was supported in part by the Army Research Laboratory (ARL) through the Cooperative Agreement (NS CTA) Number W911NF-09-2-0053, and by the Office of Naval Research (ONR) under Grant N00014-15-1-2640. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies either expressed or implied of the Army Research Laboratory or the U.S. Government.

Author contributions

All authors contributed to the study design and manuscript preparation, XL conducted simulations and contributed the proof of the properties of the regularized block model introduced here, BKS contributed the functional forms of the prior in-degree ratios. Both XL and BKS reviewed and wrote the manuscript.

Competing interests

Authors declare no conflict of interest as defined by Nature Research, or other financial and non-financial interests that might be perceived to influence the results and/or discussion reported here.

Supplementary Information

Review of degree-corrected stochastic block model

Since the standard stochastic block model makes nodes in the same block statistically indistinguishable, the most likely block assignment often places the nodes of similar degrees in a block, resulting in blocks of nodes with homogeneous degrees, too limited to represent the traditional community structures.

The degree-corrected stochastic block model [9] clusters together the nodes with heterogeneous degrees and defines the expected number of edges between nodes $i$ and $j$ as $\lambda_{ij}=\omega_{g_{i},g_{j}}\beta_{i}\beta_{j}$ where for each node $l$, $g_{l}$ is its block assignment and $\beta_{l}$ is a model parameter associated with it. This simple yet effective extension improves the performance of the models for statistical inference of block structure in the real-world networks because nodes in a community tend to have broad degree distributions. To simplify technical derivations, authors in [9] approximate the Bernoulli distribution with the Poisson. This approximation is tight when the product of the number of nodes $n$ and edge probability $p$ is small. Given a partition of the network, i.e. its block assignments $\{g_{i}\}$, reference [9] obtains the maximum likelihood estimates of the model parameters equal to $\hat{\beta}_{i}=\frac{k_{i}}{\kappa_{g_{i}}}$ and $\hat{\omega}_{rs}=m_{rs}$, where $k_{i}$ is the degree of node $i$, $\kappa_{r}$ is the sum of the degrees of all nodes in a block $r$, and $m_{rs}$ is the total number of edges between blocks $r$ and $s$, or, if $r=s$, twice the number of edges in $r$. Note that the self-loop edges and multi-edges are considered here, and the derivation is shown in Eq. 1.

Using the log-likelihood in Eq. 1 and 2 leads to a new criterion to be optimized over all possible block assignments. Ignoring the constant terms which are irrelevant to the statistical inference of $\bf g$, the log-likelihood can be expressed as

[TABLE]

where $m_{rs}$ is the total number of edges between blocks $r$ and $s$ if $r\neq s$, or twice that number otherwise, and $\kappa_{r}=\sum_{i:g_{i}=r}k_{i}$ is the sum of degrees of the nodes in block $r$. This criterion has an information-theoretic interpretation when it is presented in the alternative form

[TABLE]

which is the Kullback–Leibler divergence between the probability of observing an edge $(i,j)$ between block $r$ and $s$ in real network, i.e. $P[i\in r,j\in s]=m_{rs}/{2m}$, and the probability of observing such edge in a random graph with the same degree sequence, i.e. $P^{\prime}[i\in r,j\in s]=(\kappa_{r}/2m)(\kappa_{s}/{2m})$. Seen this way, the statistical inference maximizes the information gain when replacing the generative model of a random graph model with the degree-corrected stochastic block model. In other words, it requires the most information to describe the observed graph starting from a random graph that does not have block structure at all.

It is worth mentioning that, in the standard stochastic block model where the number of edges is drawn from the Bernoulli distribution, it is rare that $\omega_{rs}$ is close to 1. This is because the numbers of the edges between blocks are usually very small when the network is sparse. A Bernoulli random variable with a small mean is well approximated by a Poisson random variable having the same mean [18]. For this reason, [19] found no significant difference in the solutions produced by this model using the Poisson distribution compared to the original model using the Bernoulli distribution. Hence here, we refer to the standard stochastic block model as the one in which the number of edges between two nodes is drawn from the Poisson distribution.

Markov chain Monte Carlo

We use the Markov chain Monte Carlo (MCMC) algorithm [16] to infer the block assignment of the stochastic block model and its extensions. The MCMC algorithm samples the distribution of all possible block assignments in the network partitions space. Suppose there are $K$ blocks and $N$ nodes in a network. The naive Markov Chain Monte Carlo approach is not practical because the size of the partition space $O(N^{K})$ which grows exponentially in the number of blocks $K$. Therefore, Piexoto  [12] proposes the optimized MCMC algorithm with the greedy heuristic to infer the block assignment. Initially, every node in the network is assigned to one random block independently. Then, one attempts to move a node from block $r$ to $s$ with a probability conditioned on its neighbor’s block assignment $t$. The proposal function is defined as

[TABLE]

where $\epsilon>0$ is a free parameter to fulfill the ergodicity condition such that any block assignment can be reached from the current block assignment in the finite and aperiodic expected number of steps. When $\epsilon\to\infty$, the proposal function reduces to the naive scheme which assign a random block to the current node. However, in such case, the possibility of current node being assigned to the correct block assignment is very low, thus, it does not increase the log-likelihood in most cases. Consequently, such proposals are rejected very frequently, wasting the computational resource. By applying a relatively small $\epsilon$, the proposal function defined above is more likely to get accepted. The intuition behind this proposal function is that, given that a large number of edges laying across blocks $s$ and $t$, a node with many neighbors in block $t$ is likely to reside in block $s$. Consequently, the proposal function defined by Eq. 17 is more likely to be accepted, saving the computational cost for many rejected proposals.

To ensure the detailed balance, each proposed assignment is accepted with a probability $a$ in the Metropolis-Hastings fashion [17] given by

[TABLE]

where the node of the proposed assignment has $n_{t}$ neighbors in block $t$ and $\Delta\mathcal{L}$ is the change of log-likelihood after each proposal. A list of edges which are adjacent to each block are used to compute Eq. 18, which can be done in $O(k_{i})$ time [12] where $k_{i}$ is the degree of node $i$. Thus, an entire MCMC sweep of all nodes in the network requires $O(E)$ operations where $E$ is the number of edges in the networks. The mixing time of the MCMC algorithm depends on the specific input network and the starting point.

Proof of Theorem 1

Proof.

If $f_{i}=1/2$ for each $i$, then $I_{i}=O_{i}=\theta_{i}/2$ and $\sum_{i\in r}I_{i}=\sum_{i\in r}O_{i}=\sum_{i\in r}\theta_{i}/2$. Hence, $\Lambda_{rs}=\Theta_{r}/2\times\Theta_{s}/2$ where $\Theta_{r}=\sum_{i\in r}\theta_{i}$. Also, it is obvious that the sum of cross entropy on the RHS of Eq. 9 is a constant term given the block assignment $\bf g$. The log-likelihood of our model defined by Eq. 9 can be rewritten in the form

[TABLE]

where all the constant terms are grouped into the last term on the RHS. Setting the derivative of the RHS over $\theta_{i}$ to zero, we obtain $k_{i}/\theta_{i}=\sum_{s}m_{g_{i},s}/\Theta_{g_{i}}$ which means that for every node $i$ the MLE estimator $\hat{\theta}_{i}=\Theta_{g_{i}}k_{i}/\kappa_{g_{i}}$. Plugging $\hat{\theta}_{i}$ into the log-likelihood above, we obtain the log-likelihood expression in Eq. 15 after dropping all the constant terms. ∎

Proof of Theorem 2

Proof.

For given $\{f_{i}\}$, the derivative of the log-likelihood defined in Eq. 8 over $\theta_{i}$ is zero when the maximum likelihood estimator $\hat{\theta}_{i}$ satisfies the condition

[TABLE]

Even without the closed-form expression of $\hat{\theta}_{i}$, we show that Eq. 19 ensures that the expected degree of node $i$ is

[TABLE]

Since $\theta_{i}=\hat{\theta}_{i}$, we have $I_{i}=f_{i}\hat{\theta}_{i}$ and $O_{i}=(1-f_{i})\hat{\theta}_{i}$, so the RHS of Eq. 21 is actually the RHS of Eq. 19 multiplied by $\hat{\theta}_{i}$, hence it is equal to observed node degree $k_{i}$ in the network. Therefore, $\sum_{j}\lambda_{ij}=k_{i}$. ∎

Proof of Theorem 3

Proof.

We start with the equivalent definition of the generalized model in Eq. 8. The first-order derivatives of the log-likelihood over $I_{i}$ and $O_{i}$ are

[TABLE]

Given the block assignment $\bf g$, when log-likelihood achieves its maximum, the stable points of $I_{i}$ and $O_{i}$ satisfy $\frac{\partial\mathcal{L}}{\partial I_{i}}=\frac{\partial\mathcal{L}}{\partial O_{i}}=0$, which leads to the equations

[TABLE]

Suppose there are $R$ nodes in block $r$, so we have $R$ independent equations shown above and the other $R$ equations $I_{i}+O_{i}=\theta_{i}=k_{i}$ representing the assumed constraints. These $2R$ independent equations produce at most one unique set of solutions for the $2R$ variables. It is obvious that $I_{i}=k_{i}^{+}$ and $O_{i}=k_{i}^{-}$ satisfy all these $2R$ equations. Therefore, Eq. 9 can be rewritten as

[TABLE]

where $\Lambda_{rs}=\kappa_{r}^{+}\kappa_{r}^{+}$ if $r=s$; $\Lambda_{rs}=\kappa_{r}^{-}\kappa_{s}^{-}$ otherwise. Here $\kappa_{r}^{+/-}=\sum_{i\in r}{k_{i}^{+/-}}$ are the sum of internal and external degrees of nodes respectively.

The first sum on the RHS of above equation represents the Kullback–Leibler divergence between the distribution of edges residing across blocks $r$ and $s$, i.e., $p_{\text{degree}}(g_{i}=r,g_{j}=s)=\frac{m_{rs}}{2m}$ and the corresponding null distribution is defined as

[TABLE]

Note that the null model here does not require that all the edges across the entire network are randomly distributed. But it does require that, the edges inside each block and across different blocks are so distributed.

Since $\sum_{i}k_{i}=2m$, the term $\frac{k_{i}}{2m}$ can be treated as the probability of selecting an edge of node $i$. Thus, the second sum on the RHS can be treated as the expectation of binary entropy function $H(\frac{k_{i}^{+}}{k_{i}})$, i.e., $\mathbb{E}_{k_{i}}[H(\frac{k_{i}^{+}}{k_{i}})]=\sum_{i}\frac{k_{i}}{2m}(H(\frac{k_{i}^{+}}{k_{i}}))$. ∎

Real networks for experiments

Table 1 shows the number of nodes and edges of the real networks used for experiments. The number of blocks is set to their values generally accepted in the previous publications. As shown in Table 1, the Karate club network, Dolphin social network and Les Miserables characters network are relatively sparse because their average degrees range from 4 to 6. To investigate the performance of the regularized model, we add “noises” to these graph by randomly removing their edges, further increasing the sparsity of these networks. For each network, we iteratively remove random edges from the original set of edges and repeat this process twice (each time starting from the same original network). If the removal of an edge results in the isolated nodes in the network, we remove it along with its edges.

Dimensionality reduction of the partition space

In Figures 1 and 2, we project the multidimensional space of partitions into the 2D x-y plane. Specifically, for each MCMC trial, we record all the sampled partitions of the network and their corresponding log-likelihoods. Given two sampled partitions $\{g_{i}\}$ and $\{b_{i}\}$, a new partition $\{l_{i}\}$ is randomly generated such that $l_{i}=g_{i}$ with probability $0.5$ or $l_{i}=b_{i}$ otherwise. Then, we scatter all the sampled partitions along with the randomly generated partitions on the x-y plane of Figure 1. In this figure, the multiple-dimensional partition space is projected to a two-dimensional x-y plane by the multidimensional scaling (MDS) algorithm [20]. The z-axis indicates the log-likelihood of the partitions.

Illustrative toy networks

We start with an illustrative toy network to demonstrate the effect of regularization introduced here on the stochastic block model. Consider the “twin stars" network as shown in Figure 7.

There are three possible partitions of this twin stars network. The first option is to cluster the two central hubs into one block and the remaining nodes into another block. This structure corresponds to a core-periphery structure as shown in Figure 7(a) where the color of a node represents its block assignment. If we exchange the block assignments of the nodes connected to the two hubs, we obtain a twisted core-periphery structure as shown in Figure 7(b). Both of these two partitions are disassortative. For community detection problems, however, the most suitable partition should divide the network into two identical stars, which corresponds to the partition shown in Figure 7(c).

Table 2 lists the log-likelihoods of these three partitions under the standard stochastic block model (SSBM), the degree-corrected stochastic block model (DCSBM) and our block model with regularization terms (RSBM). Under the degree-corrected stochastic block model, the log-likelihood of the assortative community partition is smaller than the log-likelihood of the inappropriate twisted core-periphery partition, whereas our new model manages to identify the assortative community partition as the most appropriate one. We tested the log-likelihood of the regularized stochastic block model with the $f(k)$ function using the model parameter $\alpha=0.3$, $0.6$ and $0.9$, respectively. Note that, in Eq. 9, $\Lambda_{rs}=0$ if a block $r$ includes only degree-one nodes, causing a log-likelihood of negative infinity according to the model definition. Thus, we set the term $m_{rs}\log m_{rs}/\Lambda_{rs}=-10,000$ in Eq. 9 to ensure the log-likelihood is small enough. In general, larger $\alpha$ values would result in a larger gap in the log-likelihoods of the assortative community partition and the other two partitions. However, all the possible $\alpha$ values ensure that the assortative community partition fits the regularized model introduced here much better than the other candidates. The reason is that, no matter what value of $\alpha$ gets selected here, we have $f(1)=1$. To reduce the cross entropy terms in Eq. 14, the inference algorithm is likely to assign a degree-one node to the block in which its sole neighbor resides. Therefore, the inference algorithm is likely to return the assortative communities as the most likely partition of the network.