An Unsupervised Framework for Comparing Graph Embeddings

TL;DR

This paper introduces an unsupervised divergence score to compare graph embeddings, enabling evaluation without ground truth, by generalizing the Chung-Lu model to include geometric properties and testing on synthetic and real networks.

Contribution

It proposes a novel divergence score framework for unsupervised comparison of graph embeddings, extending the Chung-Lu model with geometric considerations.

Findings

The divergence score effectively distinguishes high-quality embeddings.

The framework works on both synthetic and real-world networks.

It provides a domain-agnostic tool for embedding evaluation.

Abstract

Graph embedding is a transformation of vertices of a graph into set of vectors. Good embeddings should capture the graph topology, vertex-to-vertex relationship, and other relevant information about graphs, subgraphs, and vertices. If these objectives are achieved, they are meaningful, understandable, and compressed representations of networks. They also provide more options and tools for data scientists as machine learning on graphs is still quite limited. Finally, vector operations are simpler and faster than comparable operations on graphs. The main challenge is that one needs to make sure that embeddings well describe the properties of the graphs. In particular, the decision has to be made on the embedding dimensionality which highly impacts the quality of an embedding. As a result, selecting the best embedding is a challenging task and very often requires domain experts. In this paper, we propose a ``divergence score'' that can be assign to various embeddings to distinguish good ones from bad ones. This general framework provides a tool for an unsupervised graph embedding comparison. In order to achieve it, we needed to generalize the well-known Chung-Lu model to incorporate geometry which is interesting on its own rights. In order to test our framework, we did a number of experiments with synthetic networks as well as real-world networks, and various embedding algorithms.

Tables

Table 1. Table 1: Kendall-tau correlation between all ranked embeddings on the College Football graph using 3 different graph clustering algorithms.

Algorithms	ECG	Louvain
Louvain	0.81
InfoMap	0.83	0.79

Equations

$$\bar{\text{c}}=(c_{1,2},\dots ,c_{1,\ell },c_{2,3},\dots ,c_{2,\ell },\dots ,c_{\ell -1,\ell })\text{and }\hat{\text{c}}=(c_1,\dots ,c_{\ell })$$

$${\bar{\text{b}}}_{\mathcal{E}}(\alpha )=(b_{1,2},\dots ,b_{1,\ell },b_{2,3},\dots ,b_{2,\ell },\dots ,b_{\ell -1,\ell })\text{ and }{\hat{\text{b}}}_{\mathcal{E}}(\alpha )=(b_1,\dots ,b_{\ell })$$

$${\Delta }_{\alpha }=\frac{1}{2}\cdot \left(JSD(\bar{\mathbf{c}},\bar{\mathbf{b}}(\alpha ))+JSD(\hat{\mathbf{c}},\hat{\mathbf{b}}(\alpha ))\right).$$

$$P(v_i,v_j)=\{\begin{array}{ll}\frac{{\mathrm{deg}}_G(v_i){\mathrm{deg}}_G(v_j)}{2|E|},& i\ne j\\ \frac{{\mathrm{deg}}_G^2(v_i)}{4|E|},& i=j.\end{array}$$

$${\mathbb{E}}_{G^{\prime }\sim \mathcal{G}(G)}[{\mathrm{deg}}_{G^{\prime }}(v_i)]$$

$$\text{w}=(w_1,\dots ,w_n)=({\mathrm{deg}}_G(v_1),\dots ,{\mathrm{deg}}_G(v_n))$$

$$d_{i,j}=\mathrm{dist}(\mathcal{E}(v_i),\mathcal{E}(v_j)).$$

$$g(d):={\left(1-\frac{d-d_{\min }}{d_{\max }-d_{\min }}\right)}^{\alpha },$$

$$d_{\min }$$

$$d_{\max }$$

$$p_{i,j}=x_i{x}_{j}g(d_{i,j})$$

$$w_i=\sum _{j\in [n]}{p}_{i,j}=x_i\sum _{j\in [n]}{x}_{j}g(d_{i,j}).$$

$$p_{i,j}^s=t_i^s{t}_j^{s}g(d_{i,j}),$$

$$s_i^s=\sum _j{p}_{i,j}^s=t_i^s\sum _j{t}_j^{s}g(d_{i,j}).$$

$$t_i^{s+1}=(1-\epsilon )t_i^s+\epsilon t_i^s(w_i/s_i^s)=t_i^s+\epsilon t_i^s(w_i/s_i^s-1).$$

$$b_i=x_i\sum _{j=1}^n{a}_{ij}{x}_j\text{ for all }i\in [n].$$

$$\sum _{i=1}^n{b}_i>2\underset{i\in [n]}{\max }{b}_i.$$

$$b_i=f_i(\mathbf{x})=f_i(\mathbf{y}),\text{ where }{f}_i(\mathbf{x})=x_i\sum _{j=1}^n{a}_{ij}{x}_j.$$

$$f_i(\mathbf{z})$$

$$\frac{df_{i}}{dh}\Big{(}\mathbf{v}+h\mathbf{s}~{}|~{}h=0\Big{)}=0.$$

$$f_i(\mathbf{v}+h\mathbf{s})=\sum _{j=1}^n{a}_{ij}(v_i+h{s}_i)(v_j+h{s}_j)$$

$$\frac{d{f}_i}{dh}(\mathbf{v}+h\mathbf{s})=\sum _{j=1}^n{a}_{ij}(s_j(v_i+h{s}_i)+s_i(v_j+h{s}_j)).$$

$$0=\frac{df_{i}}{dh}\Big{(}\mathbf{v}+h\mathbf{s}~{}|~{}h=0\Big{)}=\sum_{j=1}^{n}a_{ij}(s_{j}v_{i}+s_{i}v_{j})=s_{i}\sum_{j=1}^{n}a_{ij}v_{j}+\sum_{j=1}^{n}s_{j}a_{ij}v_{i}.$$

$${\mathbf{g}}_i\mathbf{s}=0,$$

$$G\mathbf{s}=0,$$

$$\det (C)=0,$$

$$0=1\cdot \sum _{j=2}^n{a}_{1j}{v}_1{v}_j+\sum _{j=2}^n{u}_j{a}_{1j}{v}_1{v}_j=\sum _{j=2}^n(1+u_j)a_{1j}{v}_1{v}_j.$$

$$0=a_{21}{v}_2{v}_1-\sum _{j\ne 2}{a}_{2j}{v}_2{v}_j-\sum _{j=3}^n{a}_{2j}{v}_2{v}_j=-2\sum _{j=3}^n{a}_{2j}{v}_2{v}_j.$$

$$\sum _{i=2}^n{b}_i=\sum _{i=2}^n\left(x_i\sum _{j=1}^n{a}_{ij}{x}_j\right)>\sum _{i=2}^n\left(x_i{a}_{i1}{x}_1\right)=x_1\sum _{i=1}^n{a}_{1i}{x}_i=b_1.$$

$$b_1^{\prime }=x_1^{\prime }\sum _{j\in [n]}{a}_{1j}{x}_j^{\prime }=b_1$$

$$b_i^{\prime }=x_i^{\prime }\sum _{j\in [n]}{a}_{ij}{x}_j^{\prime }=x_i^{\prime }{a}_{i1}{x}_1^{\prime }+x_i^{\prime }\sum _{j=2}^n{a}_{ij}{x}_j^{\prime }=b_1\frac{a_{1i}}{{\sum }_{j\in [n]}{a}_{1j}}+{\left(\frac{b_1}{s{\sum }_{j\in [n]}{a}_{1j}}\right)}^2\sum _{j=2}^n{a}_{ij}<b_1,$$

Full text

The main challenge is that one needs to make sure that embeddings well describe the properties of the graphs. In particular, the decision has to be made on the embedding dimensionality which highly impacts the quality of an embedding. As a result, selecting the best embedding is a challenging task and very often requires domain experts.

In this paper, we propose a “divergence score” that can be assign to various embeddings to distinguish good ones from bad ones. This general framework provides a tool for an unsupervised graph embedding comparison. In order to achieve it, we needed to generalize the well-known Chung-Lu model to incorporate geometry which is interesting on its own rights. In order to test our framework, we did a number of experiments with synthetic networks as well as real-world networks, and various embedding algorithms.

1 Introduction

The study of networks has emerged in diverse disciplines as a means of analyzing complex relational data. Indeed, capturing aspects of a complex system as a graph can bring physical insights and predictive power [1]. Network Geometry is a rapidly developing approach in Network Science [2] which further abstracts the system by modelling the vertices of the network as points in a geometric space. There are many successful examples of this approach that include latent space models [3], and connections between geometry and network clustering and community structure [4, 5]. Very often, these geometric embeddings naturally correspond to physical space, such as when modelling wireless networks or when networks are embedded in some geographic space [6, 7]. See [8] for more details about applying spatial graphs to model complex networks.

Another important application of geometric graphs is in graph embedding. The idea here is that, for a given network, one tries to embed it in a geometric space by assigning coordinates to each vertex such that nearby vertices are more likely to share an edge than those far from each other. In a good embedding most of the network’s edges can be predicted from the coordinates of the vertices. For example, in [9] protein interaction networks are embedded in low-dimension Euclidean space. Unfortunately, in the absence of a general-purpose representation for graphs, very often graph embedding requires domain experts to craft features or to use specialized feature selection algorithms. Having said that, there are some graph embedding algorithms that work without any prior or additional information other than graph structure. However, these are randomized algorithms that are usually not so stable; that is, the outcome is often drastically different despite the fact that all the algorithm parameters remain the same.

Consider a graph $G=(V,E)$ on $n$ vertices, and several embeddings of its vertices to some multidimensional spaces (possibly in different dimensions). The main question we try to answer in this paper is: how do we evaluate these embeddings? Which one is the best and should be used? In order to answer these questions, we propose a general framework that assigns the divergence score to each embedding which, in an unsupervised learning fashion, distinguishes good from bad embeddings. In order to benchmark embeddings, we generalize well-known Chung-Lu random graph model to incorporate geometry. The model is interesting on its own and should be useful for many other problems and tools. In order to test our algorithm, we experiment with synthetic networks as well as real-world networks, and various embedding algorithms.

The paper is structured as follows. In Section 2, we describe our algorithm for comparing graph embeddings, and we illustrate our approach on one simple graph. The Chung-Lu model is generalized in Section 3. Some theoretical results that justify the model can be found in Appendix A. In Section 4, we describe the experiments and present results. In particular, the datasets and embedding algorithms used are outlined respectively in subsections 4.1 and 4.2. We conclude with a discussion on some future directions in Section 5.

2 General Framework

Suppose that we are given a graph $G=(V,E)$ on $n$ vertices with the degree distribution $\textbf{w}=(w_{1},\ldots,w_{n})$ and an embedding of its vertices to $k$-dimensional space, $\mathcal{E}:V\to{\mathbb{R}}^{k}$. Our goal is to assign a “divergence score” to this embedding. The lower the score, the better the embedding is. This will allow us to compare several embeddings, possibly in different dimensions.

2.1 Intuition Behind the Algorithm

What do we expect from a good embedding? As already mentioned, in a good embedding, one should be able to predict most of the network’s edges from the coordinates of the vertices. Formally, it is natural to expect that if two vertices, say $u$ and $v$, are far away from each other (that is, $\mathrm{dist}(\mathcal{E}(u),\mathcal{E}(v))$ is relatively large), then the chance they are adjacent is smaller compared to another pair of vertices that are close to each other. But, of course, in any real-world network there are some sporadic long edges and some vertices that are close to each other are not adjacent. In other words, we do not want to pay attention to local properties such as existence of particular edges (microscopic point of view) but rather evaluate some global properties such as density of some relatively large subsets of vertices (macroscopic point of view). So, how can we evaluate if the global structure is consistent with our expectations and intuition without considering individual pairs?

The idea is as follows. We identify dense parts of the graph by running some good graph clustering algorithm. As we will illustrate in Section 4, the choice of graph clustering algorithm is flexible so long as the vertex set is partitioned into clusters such that there are substantially more edges captured within clusters than between them. The clusters that are found will provide the desired macroscopic point of view of the graph. Note that for this task we only use information about graph $G$; in particular, we do not use the embedding $\mathcal{E}$ at all. We then consider the graph $G$ from a different point of view. Using the Geometric Chung-Lu (GCL) model that we introduce in this paper especially for this purpose, based on the degree distribution w and the embedding $\mathcal{E}$, we compute the expected number of edges within each cluster found earlier, as well as between them. The embedding is scored by computing a divergence score between those expected number of edges, and the actual number of edges present in $G$. Our approach falls into a general and commonly used method of statistical inference, in our case applied to the Geometric Chung-Lu model. With these methods, one fits a generative model of a network to observed network data, and the parameters of the fit tell us about the structure of the network in much the same way that fitting a straight line through a set of data points tells us about their slope.

2.2 Algorithm

Given a graph $G=(V,E)$, its degree distribution w on $V$, and an embedding $\mathcal{E}:V\to{\mathbb{R}}^{k}$ of its vertices in $k$-dimensional space, we perform the five steps detailed below to obtain $\Delta_{\mathcal{E}}(G)$, a divergence score for the embedding. We can apply this algorithm to compare several embeddings $\mathcal{E}_{1},\dots,\mathcal{E}_{m}$, and select the best one via $\operatorname*{argmin}_{i\in[m]}\Delta_{\mathcal{E}_{i}}(G)$ (here and later in the paper, we use $[n]$ to denote the set of natural numbers less than or equal to $n$; that is, $[n]:=\{1,\ldots,n\}$). Note that our algorithm is a general framework and some parts have flexibility. We clearly identify these below.

Step 1: Run some stable graph clustering algorithm on $G$ to obtain a partition C of the vertex set $V$ into $\ell$ communities $C_{1},\ldots,C_{\ell}$.

Note: In our implementation, we used the ensemble clustering algorithm for graphs (ECG) which is based on the Louvain algorithm and the concept of consensus clustering [10], and is shown to have good stability. We experiment with other algorithms in Section 4.

Step 2: For each $i\in[\ell]$, let $c_{i}$ be the proportion of edges of $G$ with both endpoints in $C_{i}$. Similarly, for each $1\leq i<j\leq\ell$, let $c_{i,j}$ be the proportion of edges of $G$ with one endpoint in $C_{i}$ and the other one in $C_{j}$. Let

[TABLE]

be two vectors with a total of $\binom{\ell}{2}+\ell=\binom{\ell+1}{2}$ entries that sum to one. These graph vectors characterize partition C from the perspective of $G$.

Note: The embedding $\mathcal{E}$ does not affect the vectors $\bar{\textbf{c}}$ and $\hat{\textbf{c}}$. It is calculated purely based on $G$ and the partition C.

Step 3: For a given parameter $\alpha\in{\mathbb{R}}_{+}$ and the same vertex partition C, we consider $\mathcal{G}(\textbf{w},\mathcal{E},\alpha)$, the GCL Model presented in Section 3. For each $1\leq i<j\leq\ell$, we compute $b_{i,j}$, the expected proportion of edges of $\mathcal{G}(\textbf{w},\mathcal{E},\alpha)$ with one endpoint in $C_{i}$ and the other one in $C_{j}$. Similarly, for each $i\in[\ell]$, let $b_{i}$ be the expected proportion of edges within $C_{i}$. That gives us another two vectors

[TABLE]

with a total of $\binom{\ell+1}{2}$ entries that sum to one. These vectors characterizes partition C from the perspective of the embedding $\mathcal{E}$.

Note: The structure of graph $G$ does not affect the vectors $\bar{\textbf{b}}_{\mathcal{E}}(\alpha)$ and $\hat{\textbf{b}}_{\mathcal{E}}(\alpha)$; only its degree distribution w and embedding $\mathcal{E}$ are used.

Note: We used the Geometric Chung-Lu Model but the framework is flexible. If, for any reason (perhaps there are some restrictions for the maximum edge length; such restrictions are often present in, for example, wireless networks) it makes more sense to use some other model of random geometric graphs, it can be easily implemented here. If the model is too complicated and computing the expected number of edges between two parts is challenging, then it can be approximated easily via simulations.

Step 4: Compute the distances between the two pairs of vectors, between $\bar{\textbf{c}}$ and $\bar{\textbf{b}}_{\mathcal{E}}(\alpha)$, and between $\hat{\textbf{c}}$ and $\hat{\textbf{b}}_{\mathcal{E}}(\alpha)$, in order to measure how well the graph $G$ fits the model $\mathcal{G}(\textbf{w},\mathcal{E},\alpha)$. Let $\Delta_{\alpha}$ be a weighted average of the two distances.

Note: We used the well-known and widely used Jensen–Shannon divergence (JSD) to measure the dissimilarity between two probability distributions. The JSD was originally proposed in [11] and can be viewed as a smoothed version of the Kullback-Leibler divergence. In our implementation, we used simple average, that is,

[TABLE]

Other weighted averages can be used if more weight needs to be put on internal or external edges.

Step 5: Select $\hat{\alpha}=\operatorname*{argmin}_{\alpha}\Delta_{\alpha}$, and define the divergence score for embedding $\mathcal{E}$ on $G$ as: $\Delta_{\mathcal{E}}(G)=\Delta_{\hat{\alpha}}$.

Note: The parameter $\alpha$ is used to define a distance in the embedding space, as we detail in Section 3. In our implementation we simply checked values of $\alpha$ on a grid between 0 and 10. There are clearly better ways to search the space of possible values of $\alpha$ but, since the algorithm worked very fast on our graphs, we did not optimize this part.

In order to compare several embeddings for the same graph $G$, we repeat steps 3-5 above and compare the divergence scores (the lower score, the better). Let us stress again that steps 1-2 are done only once, so we use the same partition into $\ell$ communities for each embedding.

2.3 Illustration

We illustrate our framework on the well-known Zachary’s Karate Club graph (see Subsection 4.1 for the description of this and other datasets). Illustrations with other datasets are shown in Appendix B.

The parameter $\alpha\geq 0$ in the GCL model controls the distance used in the embedding space. With $\alpha=0$, the embedding is not taken into account and the classic Chung-Lu model is obtained, so only the degree distribution is accounted for. As $\alpha$ gets larger, long edges in the embedding space are penalized more severely. In the left plot of Figure 1, we show the impact of varying $\alpha$ on the two components of equation (3) which respectively consider pairs of vertices that are internal (to some cluster) or external (between clusters). Recall that the divergence score for a given embedding is obtained by choosing $\hat{\alpha}=\operatorname*{argmin}_{\alpha}\Delta_{\alpha}$. In the right plot of Figure 1, we show a 2-dimensional projection of the best embedding as obtained by our framework. The vertices are coloured according to the two known communities.

We can use the GCL model to generate edges, as with the standard Chung-Lu model. In Figure 2, we generate 3 such graphs using the best embedding shown in Figure 1. The left plot uses $\alpha=0$, which ignores the embedding and clearly generates too many long edges between the clusters. The center plot uses the optimal value ($\hat{\alpha}=2.75$ in this case), generating a graph that resembles the true one. The rightmost plot uses the larger value $\alpha=7$, which penalizes long edges more severely, yielding a graph with less edges between the two communities.

3 Geometric Chung-Lu Model

It is known that classical Erdős-Rényi (binomial) random graphs $G(n,p)$ can be generalized to $G(\textbf{w})$, the random graph with a given expected degree distribution $\textbf{w}=(w_{1},\ldots,w_{n})$. Because of our application, we will define it as a function of a given graph $G$ on $n$ vertices but, in fact, it is only a function of its degree sequence.

Since our goal is to compare different embeddings of the same graph, we will generalize the Chung-Lu model further, including geometry coming from graph embedding. In such models vertices are embedded in some metric space and link formation is influenced by the metric distance between vertices. Such models are called spatial models or geometric graphs. The main principle of spatial models is that vertices that are metrically close are more likely to link to each other. This is a formal expression of the intuitive notion we have about virtual networks: Web links are likely to point to similar pages, people that share similar interests are more likely to become friends on Facebook, and scientific papers mostly refer to papers on similar topics. See [8] for more details about applying spatial graphs to model complex networks.

3.1 Original Model

Let $G=(V,E)$ be a graph, where $V=\{v_{1},\ldots,v_{n}\}$ are the vertices, the edges $E$ are multisets of $V$ of cardinality 2 (loops are allowed), and $\deg_{G}(v)$ is the degree of $v$ in $G$ (with a loop at $v$ contributing 2 to the degree of $v$). We define $\mathcal{G}(G)$ to be the probability distribution on graphs on the vertex set $V$ following the well-known Chung-Lu model [12, 13, 14, 15]. In this model, each set $e=\{v_{i},v_{j}\}$, $v_{i},v_{j}\in V$, is independently sampled as an edge with probability given by:

[TABLE]

(Let us mention about one technical assumption. Note that it might happen that $P(v_{i},v_{j})$ is greater than one and so it should really be regarded as the expected number of edges between $i$ and $j$; for example, as suggested in [1], one can introduce a Poisson-distributed number of edges with mean $P(v_{i},v_{j})$ between each pair of vertices $i$, $j$. However, since typically the maximum degree $D$ satisfies $D^{2}\leq 2|E|$ it rarely creates a problem and so we may assume that $P(v_{i},v_{j})\leq 1$ for all pairs.)

As already mentioned, this model is a function of the degree sequence of $G$. One desired property of this random model is that it yields a distribution that preserves the expected degree for each vertex, namely: for any $i\in[n]$,

[TABLE]

3.2 Geometric Model

This time we are not only given the expected degree distribution

[TABLE]

but also an embedding of its vertices in some $k$-dimensional space, function $\mathcal{E}:V\to{\mathbb{R}}^{k}$. In particular, for each pair of vertices, $v_{i}$, $v_{j}$, we know the distance between them:

[TABLE]

It is desired for the probability that $v_{i}$ and $v_{j}$ are adjacent to be a function of $d_{i,j}$, that is, to be proportional to $g(d_{i,j})$ for some function $g$. Function $g$ should be a decreasing function as long edges should occur less frequently than short ones. There are many natural choices such as $g(d)=d^{-\beta}$ for some $\beta\in[0,\infty)$ or $g(d)=\exp(-\gamma d)$ for some $\gamma\in[0,\infty)$. We use the following, normalized function $g:[0,\infty)\to[0,1]$: for a fixed $\alpha\in[0,\infty)$, let

[TABLE]

where

[TABLE]

are the minimum, respectively maximum, distance between vertices in embedding $\mathcal{E}$. One convenient and desired property of this function is that it is invariant with respect to an affine transformation of the distance measure. Clearly, $g(d_{\min})=1$ and $g(d_{\max})=0$; in the computations, we can use clipping to force $g(d_{\min})<1$ and/or $g(d_{\max})>0$ if required. Let us also note that if $\alpha=0$ (that is, $g(d)=1$ for any $d\in[0,\infty)$ with $g(d_{\max})=0^{0}=1$), then we recover the original Chung-Lu model as the pairwise distances are neglected. Moreover, the larger parameter $\alpha$ is, the larger aversion for long edges is. Since this family of functions (for various values of the parameter $\alpha$) captures a wide spectrum of behaviours, it should be enough to concentrate on this choice but one can easily experiment with other functions. So, for now we may assume that the only parameter of the model is $\alpha\in[0,\infty)$.

The Geometric Chung-Lu (GCL) model is the random graph $G(\textbf{w},\mathcal{E},\alpha)$ on the vertex set $V=\{v_{1},\ldots,v_{n}\}$ in which each pair of vertices $v_{i},v_{j}$, independently of other pairs, forms an edge with probability $p_{i,j}$, where

[TABLE]

for some carefully tuned weights $x_{i}\in{\mathbb{R}}_{+}$. The weights are selected such that the expected degree of $v_{i}$ is $w_{i}$; that is, for all $i\in[n]$

[TABLE]

Indeed, we prove in Appendix A that there exists the unique selection of weights, provided that the maximum degree of $G$ is less than the sum of degrees of all other vertices. Since each connected component of $G$ can be embedded independently, we may assume that $G$ is connected and so the minimum degree of $G$ is at least 1. As a result, this very mild condition is trivially satisfied unless $G$ is a star on $n$ vertices. Finally, let us mention that in Appendix A it is assumed that $g(d_{i,j})>0$ for all pairs $i,j$. In our case, $g(d_{i,j})=0$ for a pair of vertices that are at the maximum distance. It causes no problems in practice but, as mentioned earlier, one can easily scale the outcome of function $g(\cdot)$ to move away from zero without affecting the divergence score in any non-negligible way.

It is not clear how to find weights explicitly but they can be easily (and efficiently) approximated numerically to any desired precision. In Appendix A, we prove that, if the solution exists, which is easy to check, then the set of right hand sides of the equations considered as a function from $\mathbb{R}^{n}$ to $\mathbb{R}^{n}$ is a local diffeomorphism everywhere in its domain. As a result, standard gradient root-finding algorithms should be quite effective in finding the desired weights. We discuss one simple numerical approximation procedure the next subsection.

3.3 Numerical Approximation

Let us start with an arbitrary vector $\textbf{t}^{0}=(t^{0}_{1},\ldots,t^{0}_{n})$ (say, $\textbf{t}^{0}=(1,\ldots,1)$) that we will carefully tune by iteratively constructing a sequence $(\textbf{t}^{s})_{s\geq 0}$. Suppose that we are given a vector $\textbf{t}^{s}=(t^{s}_{1},\ldots,t^{s}_{n})$. If we introduce an edge between $v_{i}$ and $v_{j}$ with probability

[TABLE]

then the expected degree of $v_{i}$ would be

[TABLE]

Note that, for a given vertex $v_{i}$, it easy to adjust the weights so that $s^{s}_{i}$ matches $w_{i}$, the desired expected degree of $v_{i}$; indeed, one can replace $t^{s}_{i}$ with $t^{s}_{i}(w_{i}/s^{s}_{i})$. However, unfortunately, it will also affect other values of $\textbf{s}^{s}$ and vice versa, changes in other parts of t affect $s^{s}_{i}$ too. Hence, instead of doing it, each vertex should take a small step into the right direction and this process should quickly converge to the desired state: $s^{s}_{i}$ being very close to $w_{i}$ for all $i$. Let us note that, for our application, we do not need to get close to the desired expected degree sequence. The divergence score we defined earlier is not too sensitive. Fix some small constants $\varepsilon,\delta>0$. For example, in our experiments we used $\varepsilon=0.1$ and $\delta=0.001$. For each $i$, $1\leq i\leq n$, we define

[TABLE]

We repeat the tuning process until $\max_{i}|\textbf{w}_{i}-s^{s}_{i}|<\delta$.

4 Experiments

4.1 Datasets

In order to test our algorithm, we benchmark it against synthetic graphs with communities as well as some real-world graphs with known community structures. This is a natural and often used approach (see, for example, [16]). In all examples, we colour the vertices with respect to the known, ground-truth communities, but in the algorithm, we use the partition obtained via a graph clustering algorithm, as the algorithm is unsupervised.

4.1.1 LFR

As common in the literature, we analyze how the algorithm performs on artificially constructed networks with communities. A popular example is the LFR benchmark by Lancichinetti, Fortunato, and Radicchi [17] that generates synthetic graphs that resemble real world graphs. In this benchmark, the size of each community is drawn from a power-law distribution, as is the degree of each vertex.

The LFR model has a number of parameters. The most important one is the mixing parameter $\mu$ that controls the fraction of external edges (that is, edges between communities). Essentially this can be viewed as the amount of noise in the graph. In one extreme case, if $\mu=0$, then all the edges are within communities. On the other hand, if $\mu=1$, then all edges are between different communities. Other parameters control the number of vertices ($n$), the negative exponent of the power-law degree distribution ($\gamma_{1}$), the negative exponent of the distribution of community sizes ($\gamma_{2}$), and the average ($d$) and maximum degree ($d_{\max}$).

We generated small LFR graphs with $n=100$ for visualization purpose. For the mixing parameter $\mu$, we used $\mu=0.15$ (strong communities, low noise), $\mu=0.35$ (intermediate case), and $\mu=0.55$ (noisy graph, weaker communities). For the other parameters, we used, $\gamma_{1}=2$, $\gamma_{2}=1$, $d=8$, and $d_{\max}=20$.

4.1.2 Zachary’s Karate Club

This well-known social network consists of 34 members of a karate club and 78 pairwise links observed over a period of three years. During the famous study of Zachary [18], a political conflict arose between the club president and the instructor caused the club to split into two parts (the communities), each with half of the members. Zachary recorded a network of friendships among members of the club shortly before the fission.

4.1.3 College Football

This graph represents the schedule of United States football games between Division IA colleges during the regular season in Fall 2000 [19]. This is another well-studied, more complex, real-world network with known community structures. The data consists of 115 teams (vertices) and 613 games (edges). The teams are divided into conferences containing around 8–12 teams each. In general, games are more frequent between members of the same conference than between members of different conferences, with teams playing an average of about seven intra-conference games and four inter-conference games in the 2000 season. There are a few exceptions to this rule, as detailed in [16]: one of the conferences is really a group of independent teams, one conference is really broken into two groups, and 3 other teams play mainly against teams from other conferences. We refer to those as outlying vertices, which we represent with a distinctive triangular shape.

4.1.4 Email-Eu-core Network

This network was generated using email data from a large European research institution and is available as one of the SNAP Datasets [20]. There is a directed edge $(u,v)$ in the network if person $u$ sent person $v$ at least one email. The e-mails only represent communication between institution members (the core), and the dataset does not contain incoming messages from or outgoing messages to the rest of the world. The dataset also contains community memberships of the vertices: each individual belongs to exactly one of 42 departments at the research institute.

There are 1005 vertices and 25571 edges in the original directed graph. After making it undirected and keeping only the largest connected component, we ended up with a graph on 986 vertices and 16064 edges. Most of the communities are very weak; in fact, only 1 community qualifies as a weak community as defined in equation (9.2) in [21]. Those are communities for which the ratio between the total internal degree and the total degree is above 0.5. In our study, we looked at the communities for which this ratio was the highest.

4.2 The Graph Embeddings

We considered three popular graph embedding algorithms in our examples: node2vec, VERSE and LINE. Recall that our goal is to experiment with our framework and not to do extensive comparison of graph embeddings. Comparisons are done in the original dimension of the embedding. For visualization purposes, we project the embeddings into 2-dimensional space.

4.2.1 node2vec

node2vec [22] is a semi-supervised algorithm for scalable feature learning in networks. Intuitively, the goal is to find feature representations that maximize the likelihood of preserving network neighbourhoods of vertices in a $d$-dimensional feature space. By choosing an appropriate notion of a neighborhood, node2vec can learn representations that organize vertices based on their network roles and/or communities they belong to. It is achieved by developing a family of biased random walks, which efficiently explore diverse neighborhoods of a given vertex.

There are two main parameters in this algorithm. The “return parameter” $p$ controls the likelihood of immediately revisiting a vertex in the random walk. Setting it to a high value ensures that we are less likely to sample an already-visited vertex in the following two steps (unless, of course, the next vertex in the walk had no other neighbour). The “in-out parameter” $q$ allows the search to differentiate between “inward” and “outward” vertices. Loosely speaking, this parameter guides the neighborhood sampling strategy which allows us to smoothly interpolate between BFS and DFS. For node2vec, we fixed parameter $q=1$ and we varied parameter $0.5\leq p\leq 2$. We considered embedding dimensions $2\leq D\leq 128$. For the other parameters we used the default values.

4.2.2 VERSE

VERtex Similarity Embeddings (VERSE) [23] is a simple, versatile, and memory-efficient method that derives graph embeddings explicitly calibrated to preserve the distributions of a selected vertex-to-vertex similarity measure. It is a general framework that learns any similarity measures among vertices via training a simple, yet expressive, single-layer neural network. This includes popular similarity measures such as Personalized PageRank (PPR), SimRank, and adjacency similarity. We used the default recommended parameters for this algorithm and we considered embedding dimensions $2\leq D\leq 128$.

4.2.3 LINE

Large-scale Information Network Embedding (LINE) [24] is an efficient method for vertex embedding that uses an approximate factorization of the adjacency matrix, trying to preserve first as well as second order proximities. We used the default recommended parameters for this algorithm, but we varied the number of threads used from 1 (default) to 10. We considered embedding dimensions $2\leq D\leq 128$.

4.3 Visualization

Dimension reduction seeks to produce a low dimensional representation of high dimensional data that preserves relevant structure. As a result, it is an important problem in data science as a potential pre-processing step for many machine learning algorithms. Here, we use it for visualization of tested graph embeddings that are done in high dimensions. We used UMAP (Uniform Manifold Approximation and Projection) [28], a novel manifold learning technique for dimension reduction. UMAP is constructed from a theoretical framework based in Riemannian geometry and algebraic topology. It provides a practical scalable algorithm that applies to real world datasets.

4.4 Results

4.4.1 Artificial Graphs

For the three LFR graphs described earlier, we generated 5 embeddings for every choice of parameter(s) and for each algorithm considered, for a total of 315 embeddings for each graph. In Figure 3, we plot a 2-dimension projection of the best and worst results as identified by our divergence score. The colors correspond to the ground-truth communities generated by the LFR benchmark. For the best embeddings, we clearly see that the ground-truth communities are grouped together, even (but to a lesser extent) for the noisy graph with $\mu=0.55$. This is not the case for the worst embeddings.

4.4.2 Zachary’s Karate Club Network

We already considered this dataset in Section 2. In Figure 4, we show the best and worst scoring embeddings we found using our algorithm. The colors correspond to the 2 real communities.

4.4.3 The College Football Graph

This graph consists of 115 teams spread between 12 conferences. In Figure 5, we show the best and worst scoring embeddings. The colors of vertices correspond to the conferences, white triangular shaped vertices correspond to outlying vertices as explained earlier.

For each choice of embedding model, parameters and dimension, we obtained 5 different embeddings. In Figure 6, we summarize those results by showing the mean and standard deviation of the divergence score $\Delta_{\mathcal{E}}(G)$. This allows view some overall properties: for example, embedding in 2-dimensions is not enough in this case; we obtained the best result with LINE with 32 dimensions or more. Recall that we are not performing a overall study of graph embedding algorithms, and we are not looking for a winning model. In fact, both VERSE and node2vec gave the best results for other graphs, which indicates the need for a comparison framework.

4.4.4 The email-Eu-core Graph

In Figure 7, we show the highest scoring embedding we obtained for the email graph described earlier. Since most communities are very weak, we highlight only the 3 strongest communities for which the ratio of internal to total degree are respectively 0.54, 0.42 and 0.39.

4.4.5 Graph Clustering

An important part of our algorithm consists of building a reasonable partition of the vertices, over which we build the representative vectors (1) and (2). The ECG algorithm was shown to generally yield good and stable clusters [25], and we used it throughout. We re-ran all embeddings on the College Football graph using respectively the Louvain [26] and InfoMap [27] clustering algorithms, and in each case, we produced a ranking of the embeddings with respect to our method. We compared those rankings as well as the ranking obtained with the ECG clustering algorithm by computing the Kendall-tau correlation between those. The results, which are summarized in Table 1, show very high correlation.

5 Future Directions

In this paper, our aim was to introduce a general framework for evaluating embeddings. This exploratory research showed that our algorithm is very promising. The next natural step is to do extensive experiments of various embedding algorithms on large real-world datasets in order to evaluate and compare them.

Hypergraphs are natural generalizations of graphs in which a single (hyper)edge can connect any number of vertices. As a consequence, hypergraphs are often more suitable and useful to model many important networks and processes. Typical applications are related to social data analysis and include situations such as exchanging emails with several recipients, co-authorship of research papers, or analyzing security vulnerabilities of information networks. In many situations, using hypergraphs instead of classical graphs allows us to better capture and analyze dependencies within the network. We are interested in generalizing classic notions to hypergraphs, such as clustering via modularity [29], as well as developing new algorithms for them [30]. Hence, a natural line of development of the proposed embedding algorithm is to generalize it to allow for evaluation of embeddings of hypergraphs.

As a side effect of our research on graph embeddings, we have introduced the Geometric Chung-Lu model that is interesting on its own rights and potentially applicable in other problems. As it is not the main focus of this paper, we did not analyze its graph-theoretic properties in detail. Their study remains as a subject for further research. The properties, that are standard in the analysis of other models of complex networks, include clustering coefficient, connectivity, average path length, centrality measures.

Appendix A Appendix — Geometric Chung-Lu Model is Well-defined

Let us state the problem in a slightly more general situation. Suppose that for each pair $i,j\in[n]$ we have $a_{ij}=a_{ji}\in{\mathbb{R}}_{+}$, and for each $i\in[n]$ we have $a_{ii}=0$ and $b_{i}\in{\mathbb{R}}_{+}$. In our application, elements of vector $\mathbf{b}=(b_{i})_{i\in[n]}$ satisfy $b_{i}=w_{i}\in[n-1]$ and correspond to the degree distribution, and elements of matrix $A$ satisfy $a_{i,j}=g(d_{i,j})\in(0,1]$ and correspond to the embedding distance between vertices. Our goal is to investigate if there is a solution, $x_{i}\in{\mathbb{R}}_{+}$ for $i\in[n]$, of the following system of equations:

[TABLE]

If there is one, then is this solution unique?

The $n=2$ case a degenerate case that exhibit a different behaviour but is easy to investigate: $b_{1}=x_{1}x_{2}a_{12}$ and $b_{2}=x_{2}x_{1}a_{21}$. It follows that the solution exists if and only if $b_{1}=b_{2}$. If this is the case, then there are infinite number of solutions, each of them of the form $(x_{1},x_{2})=(t,b_{1}/(a_{12}t))$ for some $t\in{\mathbb{R}}_{+}$.

Suppose now that $n\geq 3$. We will show that the desired solution of (7) exists if and only if

[TABLE]

In other words, the condition is that the maximum element in vector $\mathbf{b}$ is smaller than the sum of the other elements. This is a very mild condition that holds in our application. More importantly, this solution is unique.

We will start with proving uniqueness. After that, we will show that (8) is indeed an obvious, necessary condition before proving that it is also a sufficient one.

A.1 Uniqueness

Let us assume that $n\geq 3$. For a contradiction, suppose that we have two different solutions: $\mathbf{x}=(x_{i})_{i\in[n]}$ ($x_{i}\in{\mathbb{R}}_{+}$, $i\in[n]$) and $\mathbf{y}=(y_{i})_{i\in[n]}$ ($y_{i}\in{\mathbb{R}}_{+}$, $i\in[n]$). It follows that for all $i\in[n]$ we have

[TABLE]

Let us analyze what happens at point $\mathbf{z}=t\mathbf{x}+(1-t)\mathbf{y}$ for some $t\in[0,1]$ (that is, $z_{i}=tx_{i}+(1-t)y_{i}$, $i\in[n]$). For each $i\in[n]$ we get

[TABLE]

Note that $h_{i}^{\prime}(1/2)=0$ for all $i$ (as either $x_{i}-y_{i}$ vanishes and so $h_{i}(t)$ is a constant function or it does not vanish but then $h_{i}(t)$ is a parabola with a maximum at $t=1/2$). For convenience, let $\mathbf{v}=(\mathbf{x}+\mathbf{y})/2$ and $\mathbf{s}=(\mathbf{x}-\mathbf{y})/2$ (that is, $v_{i}=(x_{i}+y_{i})/2$ and $s_{i}=(x_{i}-y_{i})/2$ for all $i\in[n]$). It follows that

[TABLE]

On the other hand,

[TABLE]

so

[TABLE]

Combining the two observations, we get that

[TABLE]

Since $a_{ii}=0$, it is equivalent to

[TABLE]

where $\mathbf{g}_{i}$ is a row vector such that $g_{ij}=a_{ij}v_{i}$ if $i\neq j$ and $g_{ii}=\sum_{j=1}^{n}a_{ij}v_{j}$. Since this is true for all $i\in[n]$, we get

[TABLE]

where $g_{ij}$’s, the elements of matrix $G$, are defined as above. Since our assumption is that $\mathbf{x}\neq\mathbf{y}$, we have that $\|s\|>0$ and so we get that $\det(G)=0$.

Now, as multiplying a column of a matrix by a positive constant does not change the sign of its determinant, we may multiply column $j$ by $v_{j}>0$ to conclude that

[TABLE]

where $c_{ii}=\sum_{j=1}^{n}a_{ij}v_{i}v_{j}$ and $c_{ij}=a_{ij}v_{i}v_{j}$ for $i\neq j$. Thus, $C$ is a symmetric matrix whose diagonal is a sum of all its entries in a row off diagonal.

Now, since $\det(C)=0$, we get that there exists a non-zero vector $\mathbf{u}$ such that $C\mathbf{u}=\mathbf{0}$. Without loss of generality, we may assume that the first entry of this vector is a largest one in absolute value. Since we may scale this vector, without loss of generality, we may also assume that $\|\mathbf{u}\|_{\infty}=1$; that is, $u_{1}=1$ and $|u_{i}|\leq 1$ for all $i$. Let us consider the product of the $1$st row of $C$ and vector $\mathbf{u}$. We have that

[TABLE]

But this means that $u_{j}=-1$ for $j>1$ (since all other numbers involved are positive). Finally, by considering the $2$nd row of the product $C\mathbf{u}=\mathbf{0}$, we get that

[TABLE]

This is possible only if $n=2$; that is, when the sum is trivial. Since $n\geq 3$, we get the desired contradiction. This finishes the proof of the uniqueness of the solution of (7) for $n\geq 3$.

Also note that $G$ is the Jacobian matrix of the function $[f_{i}(\mathbf{x})]$ and from the proof it follows that $\det(G)\neq 0$ in all admissible points where the system of equations $b_{i}=f_{i}(\mathbf{x})$ has a solution.

A.2 Necessity

Without loss of generality, we may assume that the sequence of $b_{i}$’s is non-increasing. In particular, $b_{1}$ is a largest value. Suppose that $x_{i}\in{\mathbb{R}}_{+}$, $i\in[n]$, is a solution of the system (7). Since $a_{ij}=a_{ji}>0$ for $i\neq j$ and $a_{ii}=0$, we observe that for $n\geq 3$

[TABLE]

Hence, we get that (8) is a necessary condition for the existence of the solution.

A.3 Sufficiency

In this subsection, we will still assume that $n\geq 3$. For a contradiction, suppose that there exists a vector $\mathbf{b}=(b_{i})_{i\in[n]}$, with $b_{i}>0$ for all $i$, that satisfies (8) but for which there is no solution to the system (7). We will call such vectors infeasible; on the other hand, vectors that yield a solution $\mathbf{x}=(x_{i})_{i\in[n]}$, with $x_{i}>0$ for all $i$, will be called feasible (as proved above, the solution must be unique). Without loss of generality, we may assume that $b_{1}$ is a largest value in vector $\mathbf{b}$.

Let us now construct another vector $\mathbf{b}^{\prime}=(b^{\prime}_{i})_{i\in[n]}$ for which there is a solution to (7) (that is, $\mathbf{b}^{\prime}$ is feasible) but also $b^{\prime}_{1}=b_{1}$ is a largest element in $\mathbf{b}^{\prime}$. Indeed, it can be done easily by, for example, taking $x^{\prime}_{1}=s$ for some large enough $s\in{\mathbb{R}}_{+}$ and $x^{\prime}_{2}=\ldots=x^{\prime}_{n}=b_{1}/(s\sum_{j\in[n]}a_{1j})$. We immediately get that

[TABLE]

and for $i\geq 2$ we have

[TABLE]

provided that $s\in{\mathbb{R}}_{+}$ is sufficiently large (since $n\geq 3$, we get that $a_{1i}/\sum_{j\in[n]}a_{1j}<1$, and the second term above tends to zero as $s\to\infty$).

We will consider points along the line segment between $\mathbf{b}^{\prime}$ and $\mathbf{b}$, namely,

[TABLE]

Since $\mathbf{b}^{\prime}$ is feasible and we already proved that (8) is a necessary condition, $\mathbf{b}^{\prime}$ satisfies (8). But, not only $\mathbf{b}$ and $\mathbf{b}^{\prime}$ satisfy it but also $\mathbf{b}(t)$ satisfies it for any $t\in[0,1]$. Let us fix any $t\in(0,1)$. Clearly, $b_{1}(t)=b_{1}=b^{\prime}_{1}$ is a largest value in $\mathbf{b}(t)$. More importantly,

[TABLE]

and so, indeed, (8) holds for $\mathbf{b}(t)$. It follows that there exists a universal $\varepsilon>0$ such that for any $t\in[0,1]$ we have

[TABLE]

Fix $t\in(0,1)$ and suppose that $\mathbf{b}(t)$ is feasible. Let $\mathbf{x}(t)=(x_{i}(t))_{i\in[n]}$ be the (unique) solution for $\mathbf{b}(t)$. Let $G(t)=\left(\frac{\partial f_{i}}{\partial x_{j}}\right)_{ij}$ be the Jacobian matrix of our transformation function at point $\mathbf{x}(t)$; that is, $g_{ij}(t)=a_{ij}x_{i}(t)$ if $i\neq j$ and $g_{ii}(t)=\sum_{j\in[n]}a_{ij}x_{j}(t)$. From the analysis performed in the proof of uniqueness of the solution it follows that $\det(G(t))\neq 0$ and so our transformation is a local diffeomorphism. As a result, any open set in ${\mathbb{R}}^{n}$ containing $\mathbf{x}(t)$ is mapped to an open set in ${\mathbb{R}}^{n}$ containing $\mathbf{b}(t)$. In particular, there exists $\delta>0$ such that $\mathbf{b}(s)$ is feasible for any $t-\delta\leq s\leq t+\delta$. Combining this observation with the fact that $\mathbf{b}$ is not feasible we get that there exists $T\in(0,1]$ such that $\mathbf{b}(T)$ is not feasible but $\mathbf{b}(t)$ is feasible for any $t\in[0,T)$.

Consider then any sequence $(t_{i})_{i\in{\mathbb{N}}}$ of real numbers $t_{i}\in[0,T)$ such that $t_{i}\to T$ as $i\to\infty$; for example, $t_{i}=T(1-1/i)$. All limits from now on will be for $i\to\infty$. Recall that $\mathbf{b}(t_{i})$ is feasible and so $\mathbf{x}(t_{i})$ is well-defined.

Let us first note that it is impossible that $\|\mathbf{x}(t_{i})\|$ is bounded for infinitely many $i$. Indeed, if it were the case, then by Bolzano-Weierstrass theorem it would have a subsequence $(\mathbf{x}(t_{s_{i}}))_{i\in[n]}$ such that $\|\mathbf{x}(t_{s_{i}})\|\to c\in{\mathbb{R}}$. However, then, by continuity of our transformation, the limit value $\mathbf{b}(T)$ would be feasible. Note, in particular, that it is impossible that $x_{j}(t_{s_{i}})\to 0$ for some $j$ as then the corresponding $b_{j}(t_{s_{i}})$ would also tend to zero which implies that $b_{j}(T)=0$; this is not possible as both $b_{j}$ and $b^{\prime}_{j}$ are bounded away from zero.

It follows that $\|\mathbf{x}(t_{i})\|\to\infty$. This means that there exist $p\in[n]$ and subsequence $(t_{s_{i}})_{i\in{\mathbb{N}}}$ such that $x_{p}(t_{s_{i}})\to\infty$. Let us now observe that it is not possible that $x_{p}(t_{s_{i}})\to\infty$ and $x_{q}(t_{s_{i}})$ does not tend to zero for some $q\neq p$. Indeed, since $b_{p}(t_{s_{i}})\geq a_{pq}x_{q}(t_{s_{i}})x_{p}(t_{s_{i}})$, this would imply that for any constant $c$, $b_{p}(t_{s_{i}})>c$ for an infinite number of $i$’s. This is impossible as $b_{p}(t_{s_{i}})\to b_{p}(T)\leq b_{1}(T)$. We get that $x_{p}(t_{s_{i}})\to\infty$ for precisely one index $p$ and $x_{q}(t_{s_{i}})\to 0$ for all $q\neq p$.

Now, for a given $q\neq p$, note that

[TABLE]

and

[TABLE]

We get that for sufficiently large values of $i$, $a_{qp}x_{q}(t_{s_{i}})x_{p}(t_{s_{i}})>(1-\varepsilon/2)b_{q}(T)$, where $\varepsilon>0$ is the same as in (9). It follows that for all sufficiently large $i$,

[TABLE]

and so also in the limit

[TABLE]

But this means that

[TABLE]

that contradicts (9). Hence, we get that (8) is also a sufficient condition for the existence of the solution.

Appendix B More Illustrations of the GCL Model

In Section 2, we illustrated our framework with the small Zachary’s Karate Club graph. In this section, we provide the same illustration for all the other graphs we considered except for the larger email graph which is less amenable to nice visualization.

For each graph considered, we select the best scoring embedding we found, and we plot the divergence score as a function of the $\alpha$-parameter that we use in our framework. Recall that we search over a grid of values for $\alpha$ and we select the one for which the divergence score in (3) is minimized. We also plot the graph itself using this best embedding.

The Geometric Chung-Lu model computes probabilities of seeing edges between vertices as a function of the degrees of the corresponding vertices and their distance in embedded space, the latter being a function of $\alpha$. Thus, we can use this model to generate random graphs given the degrees of vertices and some embedding. With small values of $\alpha$, the importance of the distances is small (at the extreme case, when $\alpha=0$, it is completely neglected), and several long edges (between communities) are generated. With large values of $\alpha$, it is the opposite and we expect much fewer edges between communities. The optimal value, which we denote $\hat{\alpha}$, aims at preserving the right balance between long and short edges (respectively between and within communities).

For each graph considered, we show three generated random graph respectively using $\alpha=0,\alpha=\hat{\alpha}$, and some value of $\alpha>\hat{\alpha}$.

B.1 The College Football Graph

For this graph, most communities are very strong, and the best embedding puts their vertices quite close, as we see in Figure 8. The impact of the choice of $\alpha$ for the distance function in embedded space is clearly seen in the bottom row of Figure 8. With $\alpha=0$, there are many edges between communities, as only the expected degrees are preserved. For $\alpha$ larger than the optimal value, we see the opposite behaviour. The random graph generated with the optimal $\alpha$ value is visually very similar to the real graph shown in the top row.

B.2 LFR Graph with Strong Communities

The communities are very strong given the choice of value $\mu=0.15$, so good embeddings should put their vertices quite close, as we see in Figure 9. The impact of the choice of $\alpha$ for the distance function in embedded space is clearly seen in the bottom row of Figure 9, with the same conclusion as for the College Football graph.

B.3 LFR Graph with Weaker Communities

Results for the LFR graph with mixing parameter $\mu=0.35$ are shown in Figure 10. For the best embedding, vertices within communities are not as close as in the previous LFR graph, as expected. Conclusions regarding the values of $\alpha$ are the same as for the previous examples.

B.4 Noisy LFR Graph

In this graph, LFR graph with mixing parameter $\mu=0.55$, for each community, there are more expected external edges than internal edges. Nevertheless, in the best embedding shown in Figure 11, we still see some spatial grouping of the communities.