Detection of an arbitrary number of communities in a block spin Ising model

TL;DR

The paper introduces a method to detect multiple communities in a block spin Ising model, which can be used to understand complex systems like voting patterns and biological communities.

Contribution

The novelty lies in solving community detection for any number of groups with varying sizes and interactions, extending beyond previous two-group models.

Findings

The model can identify any number of groups with different sizes and interactions.

An explicit algorithm is provided to reconstruct the model structure from empirical correlations.

Applications include real-world voting data and biological communities.

Abstract

We study the problem of community detection in a general version of the block spin Ising model featuring M groups, a model inspired by the Curie-Weiss model of ferromagnetism in statistical mechanics. We solve the general problem of identifying any number of groups with any possible coupling constants. Up to now, the problem was only solved for the specific situation with two groups of identical size and identical interactions, see [1, 2]. Our results can be applied to the most realistic situations, in which there are many groups of different sizes and different interactions. In addition, we give an explicit algorithm that permits the reconstruction of the structure of the model from a sample of observations based on the comparison of empirical correlations of the spin variables, thus unveiling easy applications of the model to real-world voting data and communities in biology.

Figures

Equations

Full text

1 Introduction and results

In the influential article [1], Berthet, Rigollet, and Srivastava introduced the block spin Ising model consisting of two groups previously defined in the statistical mechanics literature in [3] to the field of community detection, and the authors showed that for two groups of identical size and identical coupling constants within each group the exact recovery of the community structure was possible with high probability using a sample of observations assumed to be generated by the model. The number of observations required depends on the regime (or phase) of the model. Of the three regimes of the model, the high and low temperature regimes where analysed in [1]. Subsequently, Löwe and Schubert supplied the missing critical regime in [2], and demonstrated that exact recovery was possible there, too. Thus, the problem of community detection for more than two groups and/or different coupling constants remained open. We solve the problem for a wide variety of situations as far as the number of groups in the population, the sizes of the groups, and interactions between voters within each group and between different groups are concerned, and show that reconstructing the group structure is possible given a certain number of observations. The procedure we present is completely constructive and easy to implement. Thus, our results in the present article allow for the application of community detection algorithms to real-world data. Possible applications are the identification of social classes, cultural groups, or sympathisers of political parties, or other characteristics which may not be readily available in data. Using the community detection algorithm will potentially allow social scientists to better understand societal structures derived from common preferences or interdependencies. Biologists can employ the algorithms for the problem of distinguishing species based on the observation of characteristics of different specimens. An important point that should be noted is that the techniques presented in this article can in principle also be applied to any other voting model with several communities, for which the pair correlations between votes can be deduced.

The literature on the detection of communities in stochastic models is extensive. Among the models analysed, graphical models (i.e. Markov random fields) are most prominent. See [4] for a description of how Markov random fields encode dependencies between random variables. Among graphical models, the most studied are the stochastic blockmodel and the block spin Ising model. The stochastic blockmodel (introduced in [5]) has been analysed thoroughly. See [6] for an up to date exposition and [7–10] for further important results about this model. In the stochastic blockmodel, we observe a single realisation of the random graph and assume that interactions between individuals are independent. This is very different from the block spin Ising model which is the subject of the present article.

Beyond the mathematical interest inherent to these models and the problem of community detection, there are also important applications in the field of image recognition [11,12], biology and genetics [13–15], recommendation systems [16], and natural language processing [17], among others. These applications are especially important given the ubiquitous use of large amounts of data. Other applications include the use of these models applied to problems in social sciences, such as sociological behaviour [18], migration [3], economic decision making [19–21], and political science [22]. Community detection algorithms applied to these problems allow researchers to understand the socioeconomic and political structure of societies and dependencies between different groups and classes, as well as the structures underlying species diversity and interdependence.

Compared to the stochastic blockmodel, there has been considerably less work about the community detection problem in Block spin Ising models, also referred to as multi-species mean-field models [23] or multi-group Curie-Weiss models [24–26]. In these models, there is a heterogeneous population of $[eqn]$ individuals subdivided into $[eqn]$ groups and each individual casts a binary vote in an election. We can also interpret the binary choice as a biological characteristic of the individual which can be observed and measured. The votes or measurements are random variables which are allowed to depend on each other (with different degrees of dependencies within each group and across group boundaries). These dependencies are regulated by the coupling constants between votes. We will formally present the model in Sect 1.1. An observer has access to the votes of all individuals in a number of elections, referenda, or measurements. The population is assumed static, and the observer sees the patterns in the voting behaviour. These patterns manifest in the way certain subsets of voters frequently tend to vote alike or contrary to each other. The observer’s problem consists of reconstructing the group structure of the population from the observed votes or measurements alone. The method consists in calculating the empirical pair correlations between individual votes and grouping together those voters who present the highest correlations in the sample, i.e. those who tend to agree with the highest frequency are assumed to belong to the same community. See Sects 2.2 and 2.3 for a complete description of the algorithm involved in the reconstruction procedure.

Rather than taking a single realisation of a random graph as in the stochastic blockmodel, in the block spin Ising model we observe a sample of n realisations of the model, i.e. n voting configurations of the shape $[eqn]$ which are realisations of the random vector $[eqn]$ assumed to be independent and distributed according to the probability measure of the block spin Ising model defined in Definition 1.5. Each realisation of $[eqn]$ contains the votes or measurements of the entire population in a single election, referendum, or observation. The problem we study consists of reconstructing the structure of the model in terms of assigning each of the random variables Xi, $[eqn]$ , (we will write $[eqn]$ for any $[eqn]$ ) that represents the vote of one individual to one of the M groups. Having access to a certain number of observations permits us to recover the structure with high probability for the possible structures of practical importance the model can assume.

1.1 The block spin Ising model

The block spin Ising model is defined for $[eqn]$ -valued random variables Xi indexed by $[eqn]$ , commonly referred to as spins. Since the application to voting and the observation of biological characteristics of a heterogeneous population is the main motivation of the study in this article, we will instead speak of votes and voting configurations. The voters are sorted into $[eqn]$ groups. Each group $[eqn]$ is of size $[eqn]$ , such that $[eqn]$ . The group identification function

$$[eqn]$$

assigns each voter $[eqn]$ to their respective group $[eqn]$ . Hence, the definition of the group sizes above implies $[eqn]$ , where for any countable set A the cardinality of A is denoted by $[eqn]$ .

Definition 1.1. We define the group size parameters for each group l

$$[eqn]$$

Remark 1.2. We will assume these constants $[eqn]$ , $[eqn]$ , exist and are positive. This implies in particular, that we can assume each group consists of at least two different individuals for large enough N, a fact we will use in the proofs of our results. While the population goes to infinity, the number of groups remains constant at M.

For any voting configuration $[eqn]$ , the Hamiltonian $[eqn]$ is given by

$$[eqn]$$

This Hamiltonian allows for different interactions between pairs of votes, such that if one belongs to group l and the other to group m, they interact by a coupling constant $[eqn]$ . These coupling constants subsume the inverse temperature parameter β found in the single-group Curie-Weiss model (see e.g. [27, Chapters IV and V] for a thorough discussion of the classical Curie-Weiss model). In fact, in the special case of M = 1, the definition of $[eqn]$ reduces to the Hamiltonian of the Curie-Weiss model. We note that, depending on the signs of the coupling parameters Jl,m, the value of $[eqn]$ varies with the voting configuration. In the present context of voting, we interpret $[eqn]$ as a measure of conflict in society surrounding a particular issue which gives rise to votes $[eqn]$ . If all coupling parameters are positive, there are two configurations that have the lowest possible level of conflict: the unanimous configurations $[eqn]$ and $[eqn]$ . All other configurations receive higher values $[eqn]$ . The highest levels are achieved when the votes are evenly split in each group (or closest to it in case of odd group sizes). We define the coupling matrix J to have entries equal to the constants above:

$$[eqn]$$

Definition 1.3. Let A > 0 stand for the statement that $[eqn]$ is a symmetric positive definite matrix, i.e. the Euclidean inner product $[eqn]$ is positive for all $[eqn]$ , and let $[eqn]$ mean A is positive semi-definite, i.e. $[eqn]$ holds for all $[eqn]$ .

Remark 1.4. We will assume J > 0 throughout this article. This assumption can be interpreted as the existence of stronger coupling between voters belonging to the same group versus voter coupling between groups. The assumption also allows for negative coupling between different groups, which models an antagonistic relationship between groups.

Definition 1.5. Let $[eqn]$ with J > 0 and $[eqn]$ as defined in (2). The block spin Ising model’s probability measure $[eqn]$ , which gives the probability of each of the 2^N^ voting configurations, is defined by

$$[eqn]$$

for all $[eqn]$ , where Z is a normalisation constant which depends on N and J.

1.2 Results

The model has three distinct regimes, in each of which the model behaves in a distinct fashion. This is reflected in the limiting distribution of the vector of suitably normalised sums of votes $[eqn]$ , where Sl is the sum of the votes in group $[eqn]$ . This limiting distribution is distinct in each of the three regimes. These results can be found in several articles, e.g. [23,26,28]. The three regimes are called (in adaptation of the corresponding terms for the classical single-group Curie-Weiss model) the high temperature, the critical, and the low temperature regime. The high temperature regime is characterised by the difference between the identity matrix $[eqn]$ and the coupling matrix J being a positive definite matrix, i.e. I–J > 0. The model is in the critical regime when $[eqn]$ but $[eqn]$ . Finally, the low temperature regime is equivalent to $[eqn]$ . High temperature corresponds to high disorder, meaning the voters tend to have a mind of their own with weak dependence of the votes. Low temperature corresponds to strong couplings between votes. Thus the regime depends on the social cohesion present in a society. Culturally homogeneous societies would likely fall into the low temperature regime, whereas confederations with loosely related heterogeneous groups would be expected to fall into the high temperature regime assumption.

We will study the problem of detecting the M communities in the model in the high temperature and the low temperature regimes. In the latter case, we will make an assumption (see Definition 1.8) about the Hessian matrices at the minima of the function $[eqn]$ defined by

$$[eqn]$$

which plays a crucial role in the analysis of the block spin Ising model. It appears in the de Finetti representation of the probability measure $[eqn]$ (see [26, Theorem 32] for more details). The de Finetti representation of the probabilities 1.4 as an integral over $[eqn]$ . The function F plays the role of exponentially weighting the points in the set $[eqn]$ . A smaller value of $[eqn]$ yields a higher weight in the integral. Thus, the minima of F are of particular importance, and their location depends on the regime the model is in.

Definition 1.6. We will use the symbol C for positive constants which are independent of N but may depend on the coupling matrix J and the group size parameters $[eqn]$ . We make no claim as to the precise value of these constants, and in fact they may change from one line of a calculation to the next.

Theorem 1.7. *In the high temperature regime, i.e. for I–J > 0, set H: = J^−1^−I. Then there is a positive constant $[eqn]$ such that for all $[eqn]$ *

$$[eqn]$$

*holds, where we use the notation $[eqn]$ *

This theorem is proved in Sect 3.

As in the high temperature regime, in the low temperature regime, i.e. $[eqn]$ , we address the non-critical case. Our definition of the low temperature non-critical case is similar to the corresponding definition for high temperature.

Definition 1.8 (Low temperature non-critical case). In the case that $[eqn]$ , we say that the model is in the low temperature regime. Moreover, if the Hessian of F at every point where the minimum is attained is positive definite, we say that the model is non-critical.

Lemma 1.9. In the low temperature non-critical case, the number of points where the minimum of F is attained is finite. We denote them by

$$[eqn]$$

and the corresponding Hessian of F at the point z $[eqn]$ is denoted by H_k_ and is invertible for every $[eqn]$ .

Proof: Since F is dominated by the term $[eqn]$ for large $[eqn]$ , it follows that all points where the minimum is attained must be contained in a compact set $[eqn]$ (we recall that J^−1^ is positive definite). Suppose that z is a point where the minimum is attained. The Hessian of F is positive definite at z by assumption. This implies that there is a neighbourhood $[eqn]$ of z where F attains its minimum only at z. We obtain that the set of points where the minimum is reached consists of isolated points, and it is a closed set because F is continuous. This set cannot be infinite, otherwise it would contain an accumulation point of it since B is compact, and this accumulation point would not be isolated. □

We define

$$[eqn]$$

Remark 1.10. Notice that the map

$$[eqn]$$

is bijective. We assume that there are no l and m with $[eqn]$ such that $[eqn]$ holds for every $[eqn]$ . This is equivalent to assuming that

$$[eqn]$$

The above implies that for every $[eqn]$ either $[eqn]$ or $[eqn]$ and $[eqn]$ , where the inequality derives from the non-collinearity of $[eqn]$ and $[eqn]$ and the equality condition in the Cauchy-Schwarz inequality. The vectors $[eqn]$ play a role in the group identification procedure in the high temperature regime, where it will allow us to separate the pair correlations belonging to different groups.

Theorem 1.11. *Assume the model is in the low temperature regime and non-critical as per Definition 1.8, i.e. the Hessian H_k_ of F at z_k_ is positive definite for all k. Then the function F defined in (5) has a finite number of minima, $[eqn]$ . There is a positive constant $[eqn]$ such that for all $[eqn]$ *

$$[eqn]$$

This theorem is proved in Sect 3.

The main results of this article, the proof that the community detection problem has a solution with high probability and the algorithms for the reconstruction of the group structure of the model are found in Sects 2.2 and 2.3, respectively. Theorems 1.7 and 1.11 allow us to calculate approximations for the correlations $[eqn]$ between votes for large N. We will use these approximations as a benchmark for the empirical correlations obtained from a sample of observations $[eqn]$ of voting configurations from the model:

$$[eqn]$$

(cf. Formulae (2.13) and (2.14)). From the asymptotic analysis of the block spin Ising model in Theorems 1.7 and 1.11, we know the large N value of $[eqn]$ , and by the law of large numbers the empirical correlations converge to these values as n goes to infinity. The empirical correlations also satisfy a large deviations principle (see Lemmas 2.7 and 2.21 and Proposition 2.2) which allows us to upper bound the probability of a significant deviation of the empirical correlations from these values by a function which exponentially decays to 0 with the number of observations n. Thus, with high probability, we obtain a sample of observations that are typical in that there are no large deviations of the empirical correlations from their expected values. Then we use an iterative algorithm to define an equivalence relation on the set of voters $[eqn]$ that corresponds to the underlying group structure represented by ι which is assumed to be unknown to the observer of the voting configurations $[eqn]$ . We next describe the algorithm informally for the high temperature regime (the low temperature regime algorithm is structurally similar). See the proofs of Theorems 1.12 and 1.13 for a rigorous description and Sect 1.3 for an example of the application of this algorithm.

We take a correlation

$$[eqn]$$

which according to Corollary 2.4, in the high temperature regime, corresponds to voters $[eqn]$ , who both belong to the same group, namely group $[eqn]$ with the largest value $[eqn]$ (cf. Theorem 1.7). We then identify the largest empirical correlations, which according to Lemma 2.21 belong to those voters i and j who indeed belong to said group, i.e. $[eqn]$ . Having identified the voters belonging to group l, we remove from our set of empirical correlations all those elements that correspond to voter pairs which include at least one of the indices just identified as belonging to group l. Then we pick the group $[eqn]$ with the next largest value $[eqn]$ , and repeat the last step. In each step, we identify at least one group of voters. Hence, the algorithm terminates after at most M steps. Thus, we provide an explicit algorithm of how to reconstruct the structure of the model.

We use the phrase ‘Property f holds with high probability as n goes to infinity’ in the sense that for any constant $[eqn]$ there is a constant $[eqn]$ such that for all $[eqn]$ the probability that f holds is at least $[eqn]$ .

The main theorems of this article are

Theorem 1.12 (Group Identification and Reconstruction Procedure in the High Temperature Regime). Let the model be in the high temperature regime. There is a fixed natural number $[eqn]$ and a constant $[eqn]$ (cf. Definition 2.5) such that if $[eqn]$ and $[eqn]$ , a sample of n observations $[eqn]$ allows us to recover the group partition with high probability. The probability that we cannot recover the group partition is bounded above by the exponentially decaying function

$$[eqn]$$

and

Theorem 1.13 (Group Identification and Reconstruction Procedure in the Low Temperature Regime). Let the model be in the low temperature regime and non-critical. Let $[eqn]$ be the minima of the function F, and assume there are no l and m with $[eqn]$ such that $[eqn]$ holds for every $[eqn]$ . There is a fixed natural number $[eqn]$ and a constant $[eqn]$ (cf. Definition 2.19) such that if $[eqn]$ and $[eqn]$ , a sample of n observations $[eqn]$ allows us to recover the group partition with high probability. The probability that we cannot recover it is bounded above by the following exponentially decaying function

$$[eqn]$$

Remark 1.14. The upper bounds for the probabilities that the community detection problem has a solution are tight in the sense that Sanov’s Theorem provides a corresponding lower bound as well, and said lower bound features an exponential term

$$[eqn]$$

respectively, for some $[eqn]$ in the high temperature regime and some $[eqn]$ in the low temperature regime. As such, in the general case of M groups, it is not possible to improve these bounds.

Remark 1.15. In the original articles [1,2] concerning the community detection problem in the block spin Ising model, the two groups considered where identical in all respects – group sizes and coupling constants. Therefore, community detection was only possible in terms of finding an equivalence relation on the set of voters $[eqn]$ such as those constructed in the proofs of Theorems 1.12 and 1.13. For the more general situation of several groups of arbitrary sizes and coupling constants, we can go beyond this solution and identify the equivalence classes detected and specific groups of the model. One such case is when all groups are of different sizes, i.e. $[eqn]$ for all $[eqn]$ . Another example is $[eqn]$ for all $[eqn]$ . In fact, there is a way to explicitly identify the groups in most cases. Also, the algorithm allows identifying the group structure in case there is imperfect information about the structure of the model in terms of group sizes and coupling constants. An example of the reconstruction algorithm using simulated data which illustrates that the group sizes are not strictly necessary to reconstruct the group structure of the model is included as Supplementary Material to this article.

In the next subsection, we present an example for the application of the community detection algorithm to the problem of extremist political movements. The rest of this article consists of Sect 2 where we formally state and prove the community detection results, Sect 3 in which we prove Theorems 1.7 and 1.11, and finally Sect 4 which includes some results we use in the proofs in order to make the article as self-contained as possible.

1.3 Detection of the members of extremist political parties

In this section, we describe how our community detection algorithms can be employed to detect members of political movements or parties. In many countries, extremist political parties are forbidden by the authorities. Membership is usually prohibited and heavily penalised. As such, admitting to membership risks prosecution up to lengthy prison sentences. Even if an extremist political party is not prohibited, or else it is not a formal party but more of a loose movement or association of like-minded people, it may still be the case that members do not openly identify as such. Censure from mainstream segments of the population leads to the phenomenon of self-censoring and avoidance of stating their membership openly. Thus, if we have the task of quantifying the membership numbers of extremist parties in a certain population of N people, asking for their political affiliation directly may not lead to accurate numbers, as extremist affiliations will likely be underrepresented among the responses. Instead, we take the indirect route of applying a questionnaire with n questions with binary responses. These can be yes/no questions but also questions asking about two possible solutions to current political or societal problems.

Suppose there are two political parties in a country. Party 1 is an extremist political movement shunned by sympathisers of party 2, which is a mainstream moderate party, and ‘party 3’, the remainder of the population which is not politically engaged. We endeavour to assign each respondent correctly to one of the three parties. Suppose we have a sample $[eqn]$ of n observations of voting configurations

$$[eqn]$$

obtained from the responses to our questionnaire for the entire population of $[eqn]$ voters. The assumption about the population structure implies there are three groups in our block spin Ising model (cf. Definition 1.5). The coupling matrix $[eqn]$ defined in (1.3) is assumed to be given by

$$[eqn]$$

which is positive definite. Moreover, it is easily checked that the matrix

$$[eqn]$$

is positive definite. Therefore, the model exhibits weak interactions between voters by virtue of being in the high temperature regime. We set H: = J^−1^−I (cf. Theorem 1.7), calculate

$$[eqn]$$

and order the diagonal entries of H^−1^:

$$[eqn]$$

We have

$$[eqn]$$ $$[eqn]$$

and set

$$[eqn]$$

Next, we calculate the empirical correlations between two votes for each pair $[eqn]$ , $[eqn]$ ,

$$[eqn]$$

Inspecting the empirical correlations, we notice that there is a cluster of $[eqn]$ around the value $[eqn]$ , contained in the open interval $[eqn]$ . We find that this cluster is composed of $[eqn]$ empirical correlations of the total $[eqn]$ correlations corresponding to the population as a whole. Since $[eqn]$ is the largest diagonal entry, the N1 indices i to be found in the cluster of $[eqn]$ around the value $[eqn]$ correspond to the members of group 1, i.e. members of the extremist political party. We now exclude all indices i belonging to extremists from our set of empirical correlations and thus obtain a reduced set of $[eqn]$ of $[eqn]$ . We inspect this set of correlations and find that there is a cluster of $[eqn]$ around the value $[eqn]$ , in an open interval $[eqn]$ , which contains $[eqn]$ correlations. The N2 indices to be found in these correlations belong to members of the mainstream political party. We exclude all N2 of these indices from our correlations. The remaining $[eqn]$ correlations featuring N3 indices belong to non-political members of the population. We note that the fortuitous clustering of the empirical correlations around the values $[eqn]$ is, in fact, a high probability event, provided the sample size n is large enough. (Cf. Theorems 1.12 and 1.13.) This concludes the reconstruction of this model’s group structure from the voting data in the sample.

2 Exact recovery of the group structure

In this section we prove the two main Theorems 1.12 and 1.13 of this article. We will use the Theorems 1.7 and 1.11, which are proved in Sect 3.

2.1 Large deviations theory

Here we will define the probability space on which the samples of any possible size $[eqn]$ generated by the model defined in Definition 1.5 live.

Let

$$[eqn]$$

be equipped with the σ-algebra $[eqn]$ of all subsets of $[eqn]$ . Let

$$[eqn]$$

Moreover, we define

$$[eqn]$$

provided with the σ-algebra $[eqn]$ of all subsets. We set

$$[eqn]$$

with the σ-algebra $[eqn]$ generated by the cylinder sets of the from $[eqn]$ , $[eqn]$ and $[eqn]$ .

We denote by

$$[eqn]$$

the elements of Ω. Here, $[eqn]$ for every t. We set the projections $[eqn]$ given by

$$[eqn]$$

and

$$[eqn]$$

The random vector X $[eqn]$ corresponds to observation $[eqn]$ . We define a product measure $[eqn]$ on $[eqn]$ using Kolmogorov’s extension theorem such that each X $[eqn]$ has a Curie-Weiss distribution according to Definition 1.5: for all $[eqn]$ and all $[eqn]$ ,

$$[eqn]$$

We define the random variables

$$[eqn]$$

We will next define a probability measure $[eqn]$ on Ω under which, for fixed i and j, $[eqn]$ are independent and identically distributed (i.i.d.) random variables, for fixed $[eqn]$ , $[eqn]$ are in general neither independent nor identically distributed. We identify $[eqn]$ with the random variables whose joint distribution is given by $[eqn]$ in Definition 1.5. First we define the probability measure $[eqn]$ by

$$[eqn]$$

For every $[eqn]$ , we define the product measure on $[eqn]$ by

$$[eqn]$$

Using Kolmogorov’s extension theorem, we define the measure $[eqn]$ on $[eqn]$ such that

$$[eqn]$$

for all such cylinder sets $[eqn]$ described above.

We denote by $[eqn]$ the empirical measure associated to the process $[eqn]$ , i.e.

$$[eqn]$$

We denote by $[eqn]$ the set of probability measures on $[eqn]$ , equipped with the total variation topology. Let $[eqn]$ be the set of real-valued functions on $[eqn]$ and we identify it with $[eqn]$ using the map $[eqn]$ . We provide $[eqn]$ with the topology of $[eqn]$ We use the symbol $[eqn]$ for the identity $[eqn]$ and for every $[eqn]$ and every $[eqn]$ we define

$$[eqn]$$

Therefore, we identify every measure $[eqn]$ with a continuous real-valued function on $[eqn]$ We set

$$[eqn]$$

the average of $[eqn]$ over all $[eqn]$ . We set for every $[eqn]$

$$[eqn]$$

which is a closed set that does not contain $[eqn]$ . Note that $[eqn]$ . We set

$$[eqn]$$

For every two measures, $[eqn]$ we denote by

$$[eqn]$$

the relative entropy of $[eqn]$ with respect to θ. Notice that this function $[eqn]$ is non-negative and concave as a function of $[eqn]$ , and θ is its only minimum [29, Definition 2.1.5 and the following remark].

Lemma 2.1. Let $[eqn]$ and suppose that $[eqn]$ are such that

$$[eqn]$$

It follows that

$$[eqn]$$

Proof: Set $[eqn]$ and $[eqn]$ . Inequality (2.4) implies that

$$[eqn]$$

Moreover, we have that

$$[eqn]$$

As stated above, the minimum of $[eqn]$ as a function of $[eqn]$ is $[eqn]$ . It follows that f attains its minimum at r, and $[eqn]$ . A first order Taylor series around r implies that

$$[eqn]$$

where $[eqn]$ lies between r and t, and therefore $[eqn]$ . Displays (2.6)-(2.8) imply the desired result. □

We need a very precise upper bound for the tail probabilities under $[eqn]$ . Therefore, we will use results from the proof of Sanov’s Theorem to establish said upper bound instead of taking the statement from the theorem itself. The result given in the theorem is in terms of the $[eqn]$ of a sequence in the number of observations n, but we will use an upper bound to be found in (2.9) which allows us to establish precisely how large n has to be in order to have an upper bound on the probability of atypical observations.

Proposition 2.2. For every $[eqn]$ and every $[eqn]$ , we have the upper bound

$$[eqn]$$

Proof: The result follows by the proof of Sanov’s Theorem [29, Theorem 2.1.10]. We use Eq. (2.1.12) in [29] which in our case reads as the following equation (we denote by $[eqn]$ the probability induced by $[eqn]$ ):

$$[eqn]$$

where $[eqn]$ , see above Lemma 2.1.12 in [29]. The result follows by (2.9), the definition of $[eqn]$ and Lemma 2.1, which implies

$$[eqn]$$

for every $[eqn]$ □

2.2 High temperature group identification

Lemma 2.3. Let A be a positive definite $[eqn]$ matrix. Then

$$[eqn]$$

holds.

Proof: Let el, $[eqn]$ , stand for the l-th vector of the canonical basis of $[eqn]$ . Since A is positive definite, it has a square root A^1/2^ which is itself positive definite. We have for all $[eqn]$

$$[eqn]$$ $$[eqn]$$

The strict inequality above follows from the Cauchy-Schwarz inequality, since the invertibility of A^1/2^ implies $[eqn]$ and $[eqn]$ are not linearly dependent, and thus the inequality is strict. As $[eqn]$ , we conclude that

$$[eqn]$$

□

Recall that H: = J^−1^−I>0 holds in the high temperature regime. The previous Lemma implies

Corollary 2.4. *We have for all $[eqn]$ with $[eqn]$ *

$$[eqn]$$

Using this corollary, we define the constants $[eqn]$ and $[eqn]$ , which are used in Theorem 1.12.

Definition 2.5. Let

$$[eqn]$$ $$[eqn]$$

We define

$$[eqn]$$

which is a strictly positive constant by Corollary 2.4, and

$$[eqn]$$

Moreover, we choose a fixed natural number $[eqn]$ satisfying (cf. Theorem 1.7)

$$[eqn]$$

for every $[eqn]$ .

Remark 2.6. Notice that Proposition 2.2 implies

$$[eqn]$$

In the following, we use the standard notation in which a random sample (or an observation) is denoted by lowercase letters corresponding to the random variables represented by uppercase letters. More precisely, for a $[eqn]$ , we denote by

$$[eqn]$$

We fix $[eqn]$ for the rest of this section. Given ω, we obtain a sample $[eqn]$ and calculate the empirical correlations (the realisation of (2.2) given ω)

$$[eqn]$$

for all $[eqn]$ with $[eqn]$ .

Lemma 2.7. Assume that $[eqn]$ . Let $[eqn]$ . Then we have

$$[eqn]$$

Proof: Recall $[eqn]$ and $[eqn]$ . It follows that

$$[eqn]$$

Theorem 1.7 and (2.12) imply

$$[eqn]$$

so we obtain

$$[eqn]$$

□

Let $[eqn]$ be an enumeration of the set $[eqn]$ with

$$[eqn]$$

We define a partition of the set of all groups $[eqn]$ based on the values h(u) above.

Definition 2.8. Let $[eqn]$ be a partition of $[eqn]$ induced by

$$[eqn]$$

Next we recursively define two set sequences which will be employed in the reconstruction algorithm of the group structure of the model.

Definition 2.9. Let $[eqn]$ . We define

$$[eqn]$$

and

$$[eqn]$$

Assume $[eqn]$ and $[eqn]$ have been defined for some $[eqn]$ . Set

$$[eqn]$$

and

$$[eqn]$$

Remark 2.10. It is a direct consequence of Definition 2.9 that

$$[eqn]$$ $$[eqn]$$

In what follows, we will state and prove a series of lemmas, which taken together will allow us to construct an equivalence relation on $[eqn]$ (see Proposition 2.17) based only on information contained in the sample of observations, and more specifically the empirical correlations $[eqn]$ . This equivalence relation corresponds to the partition into the sets $[eqn]$ , $[eqn]$ , and thus constitutes the solution to the community detection problem stated in Theorem 1.12.

Lemma 2.11. Let $[eqn]$ . Then we have

$$[eqn]$$

Proof: Recall that we fixed $[eqn]$ and hence $[eqn]$ is fixed as well. Suppose $[eqn]$ . By Definition 2.9,

$$[eqn]$$

holds. Lemma 2.7 supplies the upper bound

$$[eqn]$$

The two inequalities yield

$$[eqn]$$

Note that $[eqn]$ holds if and only if $[eqn]$ . By Definition 2.5,

$$[eqn]$$

and

$$[eqn]$$

The above and inequality (2.16) give

$$[eqn]$$

Thus, we have shown that $[eqn]$ implies $[eqn]$ . The converse is proved analogously. □

Lemma 2.12. Let $[eqn]$ . Suppose we have

$$[eqn]$$

for all $[eqn]$ and all $[eqn]$ . Then

$$[eqn]$$

holds for all $[eqn]$ .

Proof: Let $[eqn]$ . To obtain a contradiction, suppose $[eqn]$ . Then there is some $[eqn]$ such that $[eqn]$ or $[eqn]$ for some v < u. Without loss of generality, suppose $[eqn]$ . By assumption, this implies $[eqn]$ . However, this contradicts $[eqn]$ because of $[eqn]$ . Therefore, $[eqn]$ must hold. Since $[eqn]$ , Lemma 2.7 yields

$$[eqn]$$

and thus $[eqn]$ holds by definition of Gu:

$$[eqn]$$

□

Lemma 2.13. Let $[eqn]$ . Suppose we have

$$[eqn]$$

for all $[eqn]$ and all $[eqn]$ . Then

$$[eqn]$$

Proof: Assume $[eqn]$ . By Definition 2.9, there is no $[eqn]$ such that

$$[eqn]$$

for some v < u, as the latter would imply $[eqn]$ and thus $[eqn]$ in contradiction to our assumption.

To obtain a contradiction, assume that $[eqn]$ for some v < u. Since each group has at least two individuals (cf. Remark 1.2), there is an $[eqn]$ with $[eqn]$ . By assumption, this implies $[eqn]$ as v < u. This contradicts (2.17), and hence we conclude that $[eqn]$ . $[eqn]$ follows by the same arguments. □

Lemma 2.14. Let $[eqn]$ . Suppose we have

$$[eqn]$$

for all $[eqn]$ and all $[eqn]$ . Then

$$[eqn]$$

holds for all $[eqn]$ .

Proof: Let $[eqn]$ . By the last lemma, we have $[eqn]$ for all v < u. As $[eqn]$ , we have

$$[eqn]$$

To obtain a contradiction, assume $[eqn]$ . Then we have

$$[eqn]$$

by Definition 2.5, which contradicts (2.18). Therefore, $[eqn]$ must hold. Next assume $[eqn]$ for some w>u. Once again, Definition 2.5 implies

$$[eqn]$$

and we have a contradiction to (2.18). Hence, $[eqn]$ is proved. □

Corollary 2.15. Let $[eqn]$ . Suppose we have

$$[eqn]$$

for all $[eqn]$ and all $[eqn]$ . Then

$$[eqn]$$

holds for all $[eqn]$ .

Proof: The statement follows from Lemmas 2.12 and 2.14. □

Proposition 2.16. For all $[eqn]$ and all $[eqn]$ , we have

$$[eqn]$$

Proof: Together, Lemma 2.11 and Corollary 2.15 constitute a proof by induction over $[eqn]$ of the proposition’s statement. □

Proposition 2.17. Let the relation ∼ on $[eqn]$ be defined by

$$[eqn]$$

Then ∼ is an equivalence relation, and, for all $[eqn]$ , $[eqn]$ holds if and only if $[eqn]$ .

Proof: By Proposition 2.16, for all $[eqn]$ , the existence of a $[eqn]$ such that $[eqn]$ is equivalent to $[eqn]$ . The function $[eqn]$ yields a partition of $[eqn]$ with the equivalence classes given by $[eqn]$ , $[eqn]$ , i.e. the indices $[eqn]$ belonging to group l. Therefore, $[eqn]$ is equivalent to the statement that i and j belong to the same equivalence class given by some group $[eqn]$ . □

For the convenience of the reader, we restate Theorem 1.12. The theorem says that the problem of community detection has a solution which can be identified with high probability.

Theorem 2.18 (Group Identification and Reconstruction Procedure in the High Temperature Regime). Let $[eqn]$ and $[eqn]$ be as specified in Definition 2.5. Suppose that $[eqn]$ and n is a natural number. A sample of n observations $[eqn]$ allows us to recover the group partition of $[eqn]$ with high probability. The probability that we cannot recover the group partition is bounded above by the exponentially decaying function

$$[eqn]$$

Proof: Recall the definition of $[eqn]$ in (18) and that we have fixed $[eqn]$ . By Proposition 2.17, the set sequence Gu, $[eqn]$ , allows us to define an equivalence relation ∼ on $[eqn]$ which corresponds to the group structure of the model in the sense that for all $[eqn]$ holds if and only if i and j belong to the same group. Then the probability that recovery is impossible is bounded above by $[eqn]$ according to Remark 2.6. This concludes the proof of Theorem 1.12. □

2.3 Low temperature group identification

For the next definition, recall (1.7):

$$[eqn]$$

the notation $[eqn]$ for the minima of the function F, and Remark 1.10, and recall that we defined the subsets of Ω

$$[eqn]$$

Definition 2.19. We set

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$

(note that $[eqn]$ is positive by Remark 1.10), and

$$[eqn]$$

Moreover, we choose a fixed natural number $[eqn]$ satisfying (cf. Theorem 1.11)

$$[eqn]$$

for every $[eqn]$ .

Remark 2.20. Notice that Proposition 2.2 implies that

$$[eqn]$$

Recall that Theorem 1.11 states

$$[eqn]$$

We fix $[eqn]$ for the rest of this section. Given ω, we obtain a sample $[eqn]$ . Recall the definition of the empirical correlations

$$[eqn]$$

for all $[eqn]$ with $[eqn]$ .

Lemma 2.21. Assume that $[eqn]$ . Set $[eqn]$ . Then the following inequality holds:

$$[eqn]$$

Proof: Since $[eqn]$ , it follows that (recall that $[eqn]$ and $[eqn]$ and (2.19), (2.3))

$$[eqn]$$

for all $[eqn]$ . Recalling Theorem 1.11,

$$[eqn]$$

we obtain

$$[eqn]$$

for every $[eqn]$ . □

Let $[eqn]$ be an enumeration of the set $[eqn]$ with

$$[eqn]$$

We define a partition of the set of all groups $[eqn]$ based on the values g(u) above.

Definition 2.22. Let $[eqn]$ be a partition of $[eqn]$ induced by

$$[eqn]$$

Next we recursively define two set sequences which will be employed in the reconstruction algorithm of the group structure of the model.

Definition 2.23. Let $[eqn]$ . We define

$$[eqn]$$

and

$$[eqn]$$

Assume $[eqn]$ and $[eqn]$ have been defined for some $[eqn]$ . Set

$$[eqn]$$

and

$$[eqn]$$

Remark 2.24. It is a direct consequence of Definition 2.23 that

$$[eqn]$$ $$[eqn]$$

The proof of the next proposition follows the same steps as that of Proposition 2.17, using Lemma 2.21 instead of Lemma 2.7.

Proposition 2.25. Let the relation ∼ on $[eqn]$ be defined by

$$[eqn]$$

Then ∼ is an equivalence relation, and, for all $[eqn]$ , $[eqn]$ holds if and only if $[eqn]$ .

Theorem 2.26 (Group Identification and Reconstruction Procedure in the Low Temperature Regime). Let the model be in the low temperature regime and non-critical. Let $[eqn]$ be the minima of the function F, and assume there are no l and m with $[eqn]$ such that $[eqn]$ holds for every $[eqn]$ . There is a fixed natural number $[eqn]$ and a constant $[eqn]$ (cf. Definition 2.19) such that if $[eqn]$ and $[eqn]$ , a sample of n observations $[eqn]$ allows us to recover the group partition with high probability. The probability that we cannot recover it is bounded above by the exponentially decaying function

$$[eqn]$$

Proof: Recall the definition of $[eqn]$ in (2.19) and that we have fixed $[eqn]$ . By Proposition 2.25, the set sequence Gu, $[eqn]$ , allows us to define an equivalence relation ∼ on $[eqn]$ which corresponds to the group structure of the model in the sense that for all $[eqn]$ holds if and only if i and j belong to the same group. Then the probability that recovery is impossible is bounded above by $[eqn]$ according to Remark 2.20. This concludes the proof of Theorem 1.13. □

3 Proofs of Theorems 1.7. and 1.11

Recall that

$$[eqn]$$

We define for fixed $[eqn]$ with $[eqn]$

$$[eqn]$$

and

$$[eqn]$$

Then the representation

$$[eqn]$$

holds by Theorem 4.1 in the Appendix.

3.1 Proof of Theorem 1.7

Lemma 3.1. In the high temperature regime, where H = J^−1^−I>0, 0 is the only minimum of F and the lower bound

$$[eqn]$$

holds, where c_H_>0 is half of the minimum eigenvalue of H.

Proof: Let $[eqn]$ be the Hessian matrix of F at $[eqn]$ . A direct calculation using hyperbolic trigonometric identities shows that

$$[eqn]$$

where $[eqn]$ is the Kronecker delta. This implies that $[eqn]$ is positive definite for all $[eqn]$ . Since $[eqn]$ , 0 is the only minimum of F, and, due to $[eqn]$ , F is strictly positive on the complement of {0}. Moreover, a Taylor series expansion of the function $[eqn]$ in t of order 1 with remainder of order 2 in integral form shows that

$$[eqn]$$

where in the above equation we used $[eqn]$ and the spectral theorem which yields strict positivity of the at most M eigenvalues of the self-adjoint H. □

We restate Theorem 1.7 for the convenience of the reader:

Theorem 3.2. *In the high temperature regime, i.e. for I–J > 0, set H: = J^−1^−I. Then there is a positive constant $[eqn]$ such that for all $[eqn]$ *

$$[eqn]$$

*holds, where we use the notation $[eqn]$ *

Proof: We will show the statement for $[eqn]$ . The case $[eqn]$ can be handled analogously. For any set $[eqn]$ , let $[eqn]$ be the complement. We will work with Z2(N) and Z0(N) defined in (3.1) and (3.2). We denote by

$$[eqn]$$

the ball with radius r, centred at z, and we set (see Lemma 3.1)

$$[eqn]$$

which is selected in order to fulfil $[eqn]$ . We split Z2(N) and Z0(N) into integrals over $[eqn]$ and over $[eqn]$ . Lemma 3.1 implies that

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$

and likewise we conclude that

$$[eqn]$$

and

$$[eqn]$$

Notice that $[eqn]$ and $[eqn]$ . This implies that all even derivatives of $[eqn]$ are polynomials with even powers of $[eqn]$ and all odd derivatives of $[eqn]$ are polynomials with odd powers of $[eqn]$ . Therefore, all such derivatives are uniformly bounded on $[eqn]$ , and odd derivatives evaluated at 0 vanish (and the same holds for F). Taking this into consideration, and using that $[eqn]$ and the Hessian of F at 0 is H, the remainder formula in Taylor’s Theorem leads to

$$[eqn]$$

for some constant C, independent of x (for $[eqn]$ we use Taylor’s Theorem, for $[eqn]$ we use that F(x) and $[eqn]$ grow quadratically in $[eqn]$ ). Note that for positive s and t, $[eqn]$ , the mean value theorem implies that $[eqn]$ . This, together with the fact that $[eqn]$ is bounded and (3.7), implies that

$$[eqn]$$

Likewise, we get

$$[eqn]$$

The Taylor series of $[eqn]$ at 0 (up to second order, using that $[eqn]$ and that $[eqn]$ is uniformly bounded), implies that there is a constant C such that

$$[eqn]$$

and this leads to

$$[eqn]$$

Next, Proposition 4.2 implies that

$$[eqn]$$

It follows from (3.1), (3.4), (3.6),(3.8), (3.10), and (3.11) that

$$[eqn]$$

and it follows from (3.2), (3.5), (3.6), (3.9), and (3.11) that

$$[eqn]$$

Set

$$[eqn]$$

Then, using (3.12) and (3.13), we obtain

$$[eqn]$$

Displays (3.3) and (3.14) lead to the desired result. □

3.2 Proof of Theorem 1.11

Recall the definition of F given in (1.5) and Lemma 1.9.

Lemma 3.3. There exist constants $[eqn]$ and c > 0 such that the following estimation holds:

$$[eqn]$$

where α is the minimum value of F, and the balls $[eqn]$ are disjoint for $[eqn]$ . Also, for every k and every $[eqn]$ ,

$$[eqn]$$

Proof: The continuity of the Hessian of F shows that there are small enough numbers $[eqn]$ and $[eqn]$ such that $[eqn]$ is positive in $[eqn]$ for every k (and we take δ small enough that these balls are disjoint). A first order Taylor series expansion (with second order remainder in the integral form) for the function $[eqn]$ shows that

$$[eqn]$$

for every $[eqn]$ . For large enough $[eqn]$ , F(x) is dominated by $[eqn]$ , where c > 0 is any constant smaller than the smallest eigenvalue of J^−1^. For large enough x, there is a constant c such that $[eqn]$ . We conclude that there is a constant c > 0 such that

$$[eqn]$$

for large enough x and every k. Eqs (3.17) and (3.18) imply (3.15) for every x in the complement of a compact set $[eqn]$ not intersecting $[eqn]$ , for every k. On $[eqn]$ , $[eqn]$ attains its minimum, which we call $[eqn]$ (it cannot be smaller or equal 0 because F attains its minimum only at the points z $[eqn]$ ). On the set $[eqn]$ , (3.15) holds true as well. This is verified with the following estimation:

$$[eqn]$$

for every $[eqn]$ .

We restate Theorem 1.11 for the reader’s convenience: □

Theorem 3.4. *Assume the model is in the low temperature regime and non-critical as per Definition 1.8, i.e. the Hessian H_k_ of F at z_k_ is positive definite for all k. Then the function F defined in (1.5) has a finite number of minima, $[eqn]$ . There is a positive constant $[eqn]$ such that for all $[eqn]$ *

$$[eqn]$$

Recall the definitions of Z2(N), and Z0(N) in (3.1) and (3.2), as well as the expression

$$[eqn]$$

given in Theorem 4.1.

We set (cf. Lemma 3.3)

$$[eqn]$$

which is selected in order to fulfil $[eqn]$ (and we assume that N is large enough such that $[eqn]$ ). Lemma 3.3 implies that

$$[eqn]$$

and likewise we conclude that

$$[eqn]$$

and (see (3.16))

$$[eqn]$$

Notice that $[eqn]$ and $[eqn]$ . This implies that all even derivatives of $[eqn]$ are polynomials with even powers of $[eqn]$ and all odd derivatives of $[eqn]$ are polynomials with odd powers of $[eqn]$ . Therefore, all such derivatives are uniformly bounded in $[eqn]$ . Next, we recall that $[eqn]$ and the Hessian of F at z $[eqn]$ is Hk. A second order Taylor series expansion (with third order remainder in the mean value form) for the function $[eqn]$ shows that, for every k,

$$[eqn]$$

on $[eqn]$ . Notice that for positive s and t with $[eqn]$ we have that the mean value theorem implies that $[eqn]$ . This, together with the fact that $[eqn]$ is bounded and (3.23), implies that

$$[eqn]$$

Likewise, we get

$$[eqn]$$

We denote by

$$[eqn]$$

the second order Taylor series of $[eqn]$ at $[eqn]$ . Using that all derivatives of $[eqn]$ are uniformly bounded, Taylor’s Theorem implies that

$$[eqn]$$

and this implies that

$$[eqn]$$

Next, since the function $[eqn]$ is odd and its integral vanishes, Proposition 4.2 implies that

$$[eqn]$$

It follows from (3.1), (3.20), (3.22), (3.24), (3.26), and (3.27) that

$$[eqn]$$

and it follows from (3.2), (3.21), (3.22), (3.25), and Proposition 4.2 that

$$[eqn]$$

Eqs. (3.28) and (3.29) imply that

$$[eqn]$$

Eqs. (3.30) and (3.3) lead to the desired result.

4 Auxiliary results

Recall the definitions of Z2(N) and Z0(N) from (3.1) and (3.2) for fixed $[eqn]$ with $[eqn]$

$$[eqn]$$

and

$$[eqn]$$

The following theorem is a special case of [26, Theorem 32] a result proved for the first time in [30]. We present it here with a short proof for the reader’s convenience.

Theorem 4.1. The following identity holds

$$[eqn]$$

Proof: The Gaussian integral $[eqn]$ implies, expanding the square, that

$$[eqn]$$

Similarly, we obtain for any positive definite matrix $[eqn]$

$$[eqn]$$

the latter equality being the result of a change of variables u = N^−1^Aw. The first equality is straightforward for diagonal matrices A (using the one-dimensional version). Since A is self-adjoint, we can diagonalise it: A = U^T^DU for an orthogonal matrix U and a diagonal matrix D. A change of variables $[eqn]$ gives the desired result (using the diagonal case with Uy instead of y).

For every

$$[eqn]$$

we set

$$[eqn]$$

Recall Definition 1.5 and note that

$$[eqn]$$

Using (4.1) with A = J and $[eqn]$ and the change of variables u = N^−1^Jw, we obtain

$$[eqn]$$

Noting that the partition function Z is given by $[eqn]$ , we get

$$[eqn]$$

We finish by proving that the numerator is Z2(N) and the denominator Z0(N). The equation

$$[eqn]$$

and the associativity and commutativity of addition yield

$$[eqn]$$

Then we note that

$$[eqn]$$

since $[eqn]$ for any $[eqn]$ . Similarly,

$$[eqn]$$

and for all $[eqn]$

$$[eqn]$$

We have

$$[eqn]$$

where we again used that $[eqn]$ for all $[eqn]$ . Similarly, we obtain an expression for the denominator of (4.2):

$$[eqn]$$

The result follows by dividing numerator and denominator by 2 $[eqn]$ . □

The following result pertaining to Gaussian integrals is used repeatedly throughout the proofs. It is an immediate consequence of the definition of a multivariate normal distribution. See, e.g., [31, pp. 176–177] for a reference.

Proposition 4.2. *Fix $[eqn]$ . We have for all invertible $[eqn]$ *

$$[eqn]$$

Supporting information

S1 Text(PDF)