A close-up comparison of the misclassification error distance and the adjusted Rand index for external clustering evaluation

TL;DR

This paper compares the misclassification error distance and the adjusted Rand index to understand their differences, properties, and what they measure in clustering evaluation through theoretical analysis and simulations.

Contribution

It provides a detailed comparison of two popular clustering evaluation metrics, clarifying their properties and correcting misconceptions.

Findings

The two criteria measure different aspects of clustering quality.

Simulation results reveal distributional differences and biases.

The study clarifies the interpretation of each metric in practice.

Abstract

The misclassification error distance and the adjusted Rand index are two of the most commonly used criteria to evaluate the performance of clustering algorithms. This paper provides an in-depth comparison of the two criteria, aimed to better understand exactly what they measure, their properties and their differences. Starting from their population origins, the investigation includes many data analysis examples and the study of particular cases in great detail. An exhaustive simulation study allows inspecting the criteria distributions and reveals some previous misconceptions.

Tables

Table 1. Table 1: Confusion matrix for normal mixture clustering against the true cluster labels for the iris data set.

	Data-based labels
True labels	1	2	3
Setosa	50	0	0
Versicolor	0	48	2
Virginica	0	1	49

Table 2. Table 2: Confusion matrices for the clusterings obtained by modclust and entmerge for the DLBCL data set, as compared to the true cluster labels.

	modclust labels			entmerge labels
True labels	1	2	3	1	2	3	4	5
$A$	47	197	7	16	7	0	14	214
$B$	0	1408	153	0	146	929	417	69
$C$	0	278	1216	0	1191	81	63	159
$D$	0	62	0	0	0	0	0	62
$E$	4813	2	0	4809	0	0	1	5

Table 3. Table 3: Confusion matrix for Steinley’s example.

	Clustering $𝒟$
Clustering $𝒞$	$D_1$	$D_2$	$D_3$	$D_4$	$D_5$
$C_1$	1	0	1	1	0
$C_2$	0	1	0	0	1
$C_3$	1	0	1	0	1
$C_4$	0	1	0	1	0
$C_5$	1	0	1	0	1

Equations

$$d_{\mathrm{M}}(\mathcal{C},\mathcal{D})=\frac{1}{2}\underset{\sigma \in {\mathcal{P}}_s}{\min }\sum _{i=1}^{s}P(C_i△D_{\sigma (i)}),$$

$$d_{\mathrm{H}}(\mathcal{C},\mathcal{D})=\sum _{i=1}^r{P}^2(C_i)+\sum _{j=1}^s{P}^2(D_j)-2\sum _{i=1}^r\sum _{j=1}^s{P}^2(C_i\cap D_j).$$

$$\widehat{d}_{\rm M}(\mathscr{C},\mathscr{D})=\frac{1}{2}\min_{\sigma\in\mathcal{P}_{s}}n^{-1}\sum_{i=1}^{s}\big{\{}n_{i+}+n_{+\sigma(i)}-2n_{i,\sigma(i)}\big{\}}=1-n^{-1}\max_{\sigma\in\mathcal{P}_{s}}\sum_{i=1}^{r}n_{i,\sigma(i)},$$

$$\widehat{d}_{\rm H}(\mathscr{C},\mathscr{D})=n^{-2}\Big{\{}\sum_{i=1}^{r}n_{i+}^{2}+\sum_{j=1}^{s}n_{+j}^{2}-2\sum_{i=1}^{r}\sum_{j=1}^{s}n_{ij}^{2}\Big{\}},$$

$$\mathrm{RD}(\mathcal{C},\mathcal{D})={\left(\genfrac{}{}{0ex}{}{n}{2}\right)}^{-1}\left\{\sum _{i=1}^r\left(\genfrac{}{}{0ex}{}{n_{i+}}{2}\right)+\sum _{j=1}^s\left(\genfrac{}{}{0ex}{}{n_{+j}}{2}\right)-2\sum _{i=1}^r\sum _{j=1}^s\left(\genfrac{}{}{0ex}{}{n_{ij}}{2}\right)\right\}$$

$$\mathrm{ARD}(\mathcal{C},\mathcal{D})=1-\mathrm{ARI}(\mathcal{C},\mathcal{D})=\mathrm{RD}(\mathcal{C},\mathcal{D})/E(\mathrm{RD}),$$

$$a$$

$$b$$

$$c$$

$$d$$

$$\mathrm{ARD}(\mathcal{C},\mathcal{D})=\left(\genfrac{}{}{0ex}{}{n}{2}\right)(b+c)/\{(a+b)(b+d)+(a+c)(c+d)\}.$$

$$\mathbf{N}=\left(\begin{array}{cc}0& n_{12}\\ n_{21}& 0\end{array}\right).$$

$$\mathrm{MED}$$

$$\mathrm{RD}$$

$${\mathbf{N}}_1=\left(\begin{array}{cc}16& 2\\ 2& 0\end{array}\right),{\mathbf{N}}_2=\left(\begin{array}{cc}11& 0\\ 4& 5\end{array}\right).$$

$$\mathbf{N}=\left(\begin{array}{cc}{d}_1& d_2/2\\ d_2/2& 0\end{array}\right),\text{or}\mathbf{N}=\left(\begin{array}{cc}{d}_1& (d_2-1)/2\\ (d_2+1)/2& 0\end{array}\right),$$

$${\alpha }_n(d_1)=\frac{4n(n-1)d_1}{(d_1+n)\{d_1^2+n(n-2)\}}$$

$${\mathbf{N}}=\left(\begin{array}[]{cccc}q_{1}&q_{2}&\cdots&q_{s}\\ 1&0&\cdots&0\\ \vdots&\vdots&\ddots&\vdots\\ 1&0&\cdots&0\\ \end{array}\right),$$

$$n(n-1)\max \mathrm{RD}=(n-r+1{)}^2+(r-1)(2r-3)-s{k}^2-l(2k+1).$$

$$\mathbf{N}=\left(\begin{array}{cccc}{p}_1& q_2& \cdots & q_s\\ p_2& 0& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ p_r& 0& \cdots & 0\end{array}\right),$$

$${\mathbf{N}}_1=\left(\begin{array}{ccc}4& 4& 4\\ 4& 4& 4\end{array}\right)\text{and}{\mathbf{N}}_2=\left(\begin{array}{ccc}15& 4& 1\\ 4& 0& 0\end{array}\right)$$

$${\mathbf{N}}_3=\left(\begin{array}{ccc}3& 3& 3\\ 3& 3& 3\\ 3& 3& 3\end{array}\right)\text{and}{\mathbf{N}}_4=\left(\begin{array}{ccc}15& 5& 1\\ 5& 0& 0\\ 1& 0& 0\end{array}\right)$$

$${\mathbf{N}}_1=\left(\begin{array}{ccc}8& 0& 0\\ 0& 6& 0\\ 0& 0& 6\end{array}\right),{\mathbf{N}}_2=\left(\begin{array}{ccc}3& 2& 3\\ 2& 2& 2\\ 2& 2& 2\end{array}\right),{\mathbf{N}}_3=\left(\begin{array}{ccc}1& 2& 5\\ 3& 1& 2\\ 2& 4& 0\end{array}\right).$$

$$\mathbf{N}=\left(\begin{array}{ccc}16& 1& 1\\ 1& 0& 0\\ 1& 0& 0\end{array}\right),$$

$$\underset{\sigma \in {\mathcal{P}}_s}{\min }\sum _{i=1}^s{m}_{i,\sigma (i)},$$

$$M(q_1,\dots ,q_s)=2(r-1)q_1-\sum _{j=1}^s{q}_j^2+c,$$

Full text

1 Introduction

The adjusted Rand index (ARI) introduced in Hubert and Arabie, (1985) is one of the most commonly used measures of performance for clustering evaluation. Indeed, it was the recommended choice in the seminal paper of Milligan and Cooper, (1986), where five criteria were examined regarding the task of comparison of hierarchical clustering algorithms across different hierarchy levels. Their recommendation is based on the fact that, for the null case data (i.e., for a synthetic sample with randomly assigned class labels, showing no significant cluster structure), the ARI was the only index that produced a flat response curve across hierarchy levels, with mean values close to zero, hence indicating that the agreement between the randomly assigned labels and the algorithm solution was due to chance.

Another popular measure for clustering validation, not included in Milligan and Cooper’s study, is the misclassification error distance (MED). Its first appearance in the literature dates back at least to Régnier, (1965), where it was introduced as a distance between partitions of a finite set, and it was called transfer distance. It is also referred to as partition distance (Gusfield,, 2002) or maximum matching distance (Rossi,, 2015). Many papers concerning clustering evaluation indeed contain detailed comparisons of both, the ARI and the MED, showing arguments in favour of one or the other; see, for instance, Steinley, (2003, 2004), Denœud and Guénoche, (2006) or Meilă, (2005, 2007, 2016).

Whereas Steinley, (2004) supports Milligan and Cooper’s recommendation by inspecting the performance of the ARI and the MED on an exhaustive simulation study, Meilă, (2016) suggests that the MED “comes closest to satisfying everyone” in terms of its properties and ease of interpretation, Denœud and Guénoche, (2006) suggest that the MED is much appropriate for small sample sizes from their study of all the clusterings at a close number of transfers from a given one, and von Luxburg, (2010) considers the MED as “the most convenient choice from a theoretical point of view”.

It must be stressed that both criteria are commonly categorized as “external”, in the sense that they are used to measure the performance of a data-based clustering algorithm against a true cluster structure, known in advance in a simulation scenario or after a data inspection by an expert, which is taken as the ideal clustering solution, but is external to the clustering methodology itself. Internal criteria (such as those based on cohesion, entropy, cluster separation, etc) are also frequently used, but they will not constitute the focus of this paper; see Hennig, (2019) for a thorough review of internal cluster validation indexes.

This paper aims to provide further comparisons between the MED and the ARI, at several levels. Indeed, many other external criteria could be considered as well, and they are also reviewed in the aforementioned comparative studies, but here the discussion is restricted to the former two because they are usually recognized as the main criteria used in practice. The close-up inspection examines a wide range of features: Section 2 first glances through their population origins (i.e., their counterparts in the case where the true underlying data distribution is fully known) and then elaborates on their traditional, and more common, data-based versions. The comparison of these empirical analogues is the subject of Sections 3 (theoretically) and 4 (by simulations). The theoretical study comprises their computation, some illustrations by means of simple examples, and an analysis of their extreme values in relation to the case of independent clusterings. The simulation scenarios investigate the distributions of the criteria in the null case and how they evolve as two clusterings become apart from perfect agreement. Finally, Section 5 discusses the new findings and their implications.

2 Population and empirical distances between clusterings

2.1 The population version of cluster analysis

Cluster analysis is mostly posed as a sample problem, and perhaps that is one of the reasons why many authors have called attention to the lack of theoretical results for clustering (Milligan,, 1996, von Luxburg and Ben-David,, 2005), as opposed to regression or classification, where the population background is much more clearly established.

Traditionally, the goal of clustering techniques is to provide a partitioning of a data set into groups. For that goal, it suffices to have an algorithm which is appropriate for the data set at hand. However, from a statistical perspective, such a given data set is not simply a set of points in the space, but a sample from some probability distribution $P$. Hence, the goal of clustering methodologies can not reduce to partitioning only the data set at hand, but it must provide a mechanism to assign group labels to any point in the space; or, at least, to all the points in the sample space, since they could have been equally drawn as sample points. Such a view of clustering is shared by many authors, including Györfi et al., (2002, p. 245), Ben-David, von Luxburg and Pál, (2006), Klemelä, (2009, p. 196), Chacón, (2015) or Wasserman, (2018, Section 2.3).

Hence, the object that clustering algorithms should produce is not only a partition of the data set, but a whole-space partition. This means that if $\Omega$ denotes the sample space, a whole-space clustering is a class of sets $\mathscr{C}=\{C_{1},\dots,C_{r}\}$ such that $C_{i}\cap C_{j}=\varnothing$ for all $i\neq j$ and $C_{1}\cup\cdots\cup C_{r}=\Omega$. Indeed, most existing clustering methodologies are able to produce this type of object; this is the case, for instance, for $K$-means clustering, modal clustering or mixture model clustering (see Chacón,, 2015). Obviously, any partition of $\Omega$ induces a partition of the observed data set as well. To avoid confusions, these are referred to as a whole-space clustering and a clustering of the data, respectively. Also, note that both objects can have a population version (the partition that would be made if the true underlying distribution were fully known) and a data-based version (the partition that would be made after observing the data).

That made clear, to evaluate the performance of clustering methods from a statistical point of view it is necessary to employ a distance between whole-space clusterings. While there exist many notions of distance between partitions of a finite set (Day,, 1981, Meilă,, 2016), proposals to serve as a distance between whole-space clusterings do not abound in the literature. Two of them are described next.

First, since the parts of a clustering (i.e., the clusters) are sets, it seems natural for distances between clusterings to be built upon a notion of discrepancy between sets. A usual way to express the discrepancy between two sets $C$ and $D$ is by quantifying the content of their symmetric difference $C\triangle D$. This difference is defined as the elements that $C$ and $D$ do not have in common; that is, $C\triangle D=(C\cup D)\setminus(C\cap D)$. Then, taking into account the distinctive features of a partition, this natural distance between sets can be extended to define a distance between two clusterings $\mathscr{C}=\{C_{1},\dots,C_{r}\}$ and $\mathscr{D}=\{D_{1},\dots,D_{s}\}$, by adding up the contributions of the regions that their most similar clusters do not have in common. Specifically, Chacón, (2015) defined the distance in measure between $\mathscr{C}$ and $\mathscr{D}$ as

[TABLE]

where $\mathcal{P}_{s}$ is the set of all permutations of $s$ elements and, without loss of generality, it is assumed that $r\leq s$ so that $\mathscr{C}$ would be enlarged by adding $s-r$ empty sets $C_{r+1}=\cdots=C_{s}=\varnothing$ if necessary. More intuitively, $d_{\rm M}(\mathscr{C},\mathscr{D})$ represents the minimum probability mass that needs to be moved (or re-labeled) to transform $\mathscr{C}$ into $\mathscr{D}$, or viceversa.

The above $d_{\rm M}(\mathscr{C},\mathscr{D})$ is a clustering distance, in the sense of Ben-David, von Luxburg and Pál, (2006, Definition 3). Nevertheless, these authors considered a different distance between whole-space clusterings, $d_{\rm H}(\mathscr{C},\mathscr{D})$, which they called Hamming distance. This second distance is more closely related to the Rand index (as detailed below), since it is defined as the probability that two independent random observations (drawn from $P$) belong to the same cluster with respect to one of the clusterings and to different clusters with respect to the other clustering. Hence, it can be shown that an explicit expression for this Hamming distance is

[TABLE]

The dependence of this measure on squared probabilities may appear somehow unnatural, but it is a consequence of the fact that it is based on comparing the cluster labels of pairs of points.

2.2 Comparing two clusterings of the data

In a simulation setting, where the true underlying distribution $P$ is fully known, it is possible to compute the ideal population clustering; that is, the whole-space partition that would be made on the basis of this knowledge of $P$ (this ideal partition varies from one methodology to another, depending on the notion of cluster that they seek after). Hence, it is natural to evaluate the performance of a clustering technique by means of the distance from the produced data-based clustering to its population counterpart. Since both are clusterings of the whole space, any of the previously mentioned distances between whole-space clusterings can be employed.

Of course, things are different when dealing with real data. Suppose that we have observed $n$ data points $\mathcal{X}=\{x_{1},\dots,x_{n}\}$. Even if the usual methods are able to produce whole-space clusterings with the sole information provided by $\mathcal{X}$, the fact that a clustering distance depends on $P$ (Ben-David, von Luxburg and Pál,, 2006), which is unknown for real data sets, implies that to compute the clustering distance in practice it is necessary to replace $P$ by the empirical distribution $P_{n}$, which assigns probability mass $1/n$ to each data point. This means that only the labels of the data points are used in the comparison between the two clusterings, so that a distance between whole-space clusterings becomes in fact a distance between two clusterings of the data.

When this reasoning is applied to the two distances in the previous section, it results in two well-known distances between partitions of a finite set. To see this, given two partitions $\mathscr{C}=\{C_{1},\dots,C_{r}\}$ and $\mathscr{D}=\{D_{1},\dots,D_{s}\}$ of $\mathcal{X}$, with $r\leq s$, denote by $n_{ij}$, $n_{i+}=\sum_{j=1}^{s}n_{ij}$ and $n_{+j}=\sum_{i=1}^{r}n_{ij}$ the cardinalities of $C_{i}\cap D_{j}$, $C_{i}$ and $D_{j}$, respectively. The $(r\times s)$-matrix $\mathbf{N}=(n_{ij})$ is known as the confusion matrix (or contingency table), and the vectors $(n_{1+},\dots,n_{r+})$ and $(n_{+1},\dots,n_{+s})$ constitute its row-wise and column-wise margins, respectively. Then, taking into account that $P(C_{i}\triangle D_{j})=P(C_{i})+P(D_{j})-2P(C_{i}\cap D_{j})$, it follows that the empirical version of the distance in measure (1) is

[TABLE]

which coincides with the definition of the misclassification error distance (see Meilă,, 2005), so that it will be denoted as ${\rm MED}(\mathscr{C},\mathscr{D})=\widehat{d}_{\rm M}(\mathscr{C},\mathscr{D})$ henceforth (or simply MED, if it is obvious which clusterings are being compared). The MED inherits from its population version a clear interpretation as the minimum proportion of data points that would need to be re-labeled so that $\mathscr{C}$ and $\mathscr{D}$ coincided, and that is why it is also known as transfer distance (Régnier,, 1965).

On the other hand, the empirical equivalent of the Hamming distance (2) is

[TABLE]

which is also known as equivalence mismatch coefficient (Mirkin and Chernyi,, 1970; Mirkin,, 1996, p. 241) or as $n$-invariant Mirkin metric (Meilă,, 2016). Being a sample equivalent of (2), $\widehat{d}_{\rm H}(\mathscr{C},\mathscr{D})$ equals the proportion of pairs $\{(x_{k},x_{l})\colon k,l=1,\dots,n\}$ that belong to the same cluster in one of the clusterings and to different clusters in the other clustering. Note that, somehow artificially, this empirical version of the Hamming distance is taking into account data pairs of type $(x_{k},x_{k})$ as well.

In statistical terms, if $X,Y$ are independent random variables with distribution $P$ and we denote by $I_{A}$ the indicator function of a set $A$, the squared probability $P^{2}(C_{i})=E[I_{C_{i}}(X)I_{C_{i}}(Y)]$ appearing in (2) is estimated in (3) by the observed value of the $V$-statistic $n^{-2}\sum_{k,l=1}^{n}I_{C_{i}}(x_{k})I_{C_{i}}(x_{l})=n_{i+}^{2}/n^{2}=P_{n}^{2}(C_{i})$. However, $U$-statistics theory (Lee,, 1990) shows that a better estimate of $P^{2}(C_{i})$ is ${n\choose 2}^{-1}\sum_{1\leq k<l\leq n}I_{C_{i}}(x_{k})I_{C_{i}}(x_{l})={n\choose 2}^{-1}{n_{i+}\choose 2}$. Reasoning similarly for the other terms in (2) and making these changes everywhere in (3) yields the definition of the Rand distance

[TABLE]

which equals the proportion of unordered data pairs $\mathcal{U}=\{\{x_{k},x_{l}\}\colon k,l=1,\dots,n,\,k\neq l\}$ that belong to the same cluster in one of the clusterings and to different clusters in the other clustering (see Filkov and Skiena,, 2004). This distance was called symmetrical difference distance in Denœud and Guénoche, (2006), and it is also considered in Azizyan et al., (2015), under the denomination pairwise clustering loss. In any case, it is not hard to check that ${\rm RD}(\mathscr{C},\mathscr{D})=\frac{n}{n-1}\widehat{d}_{\rm H}(\mathscr{C},\mathscr{D})$, so in fact there is little difference between these two empirical versions of $d_{\rm H}$.

Instead of measuring the dissimilarity between clusterings using a distance, clustering comparisons can be based on indices that quantify the agreement between them, with values close to 1 indicating greater similarity. In this sense, the Rand index (Rand,, 1971) is defined as ${\rm RI}(\mathscr{C},\mathscr{D})=1-{\rm RD}(\mathscr{C},\mathscr{D})$. An important feature of the RI is that it also has a clear interpretation as the proportion of unordered data pairs that either belong to the same cluster or to different clusters in both clusterings. However, Fowlkes and Mallows, (1983) noted that, when comparing two clusterings with $r=s$, the range of possible values of the RI is quite narrow and its expected value $E({\rm RI})$ quickly approaches 1 as $r=s\to n$. This expectation is meant with respect to a random choice of the entries of the confusion matrix, while keeping its margins fixed, intended to reproduce a null scenario corresponding to independent clusterings. To amend this problem, Hubert and Arabie, (1985) proposed to correct the RI for chance, so that it yields an expected value of zero in such a null scenario, and introduced the adjusted Rand index ${\rm ARI}(\mathscr{C},\mathscr{D})=\{{\rm RI}(\mathscr{C},\mathscr{D})-E({\rm RI})\}/\{1-E({\rm RI})\}$. Milligan and Cooper, (1986) showed that, in addition, this correction also results in a much wider range of possible values for the ARI over the RI.

The most notable loss along this correction is the interpretation; for example, it is not easy to discern what an ARI value of $0.78$ means, or if an ARI of $0.82$ for two clusterings denotes a higher agreement between them than an ARI of $0.73$ for a different pair of clusterings, since the baseline $E({\rm RI})$ could be different. Precisely, in a series of papers (later collected in a single volume), Goodman and Kruskal, (1979) emphasized the importance for association measures in cross classifications to have a clear operational interpretation. Besides, Wallace, (1983) raised some doubts with respect to the choice of the null scenario in the computation of $E({\rm RI})$ and, more recently, Gates and Ahn, (2017) showed that the use of different null models for index adjustment can lead to disparate conclusions.

Despite these drawbacks, the ARI is one of the most popular and employed indicators for clustering comparison, in close competition with the MED. Hence, one of the main contributions of this paper is to provide a detailed inspection of both of them, via simple examples to help understanding their behaviour and their differences. Additional references concerning deep investigation of these criteria include Warrens, (2008), Steinley, Brusco and Hubert, (2016) and Steinley and Brusco, (2018), in the case of the ARI, and Charon et al., (2006), Charon, Denœud and Hudry, (2007) and Denœud, (2008) regarding the MED.

Here, since the MED is a distance and the ARI is an index, to facilitate their comparison the ARI will be previously transformed into a distance, which will be called the adjusted Rand distance and is defined as

[TABLE]

that is, as the Rand distance normalized by its expected value under the null model. This way, the ARD has unit expected value under the null model.

3 Detailed comparison of the MED and the ARD

In the following, several aspects of the MED and the ARD will be compared in detail. First, explicit computation of the two criteria is addressed. Then, the differences between the two are illustrated through several specific examples. Next, an exhaustive study of the simplest case of a $2\times 2$ confusion matrix is provided, with emphasis on exploring the most dissimilar situation between two clusterings. Finally, some of the lessons learned from the $2\times 2$ case are generalized for two clusterings of arbitrary size.

3.1 Computation

One undeniable advantage of the ARD over the MED is its simpler definition, which readily translates into a much simpler computation.

Indeed, let us write $x_{k}\sim_{\mathscr{C}}x_{l}$ if the data points $x_{k}$ and $x_{l}$ belong to the same cluster in $\mathscr{C}$ (and $x_{k}\nsim_{\mathscr{C}}x_{l}$ otherwise), and consider the cardinalities of the sets of (unordered) data pairs that cover all the possibilities of belonging either to the same or to different clusters in $\mathscr{C}$ and $\mathscr{D}$, denoted

[TABLE]

Then, it is clear that ${\rm RD}(\mathscr{C},\mathscr{D})={n\choose 2}^{-1}(b+c)$. Moreover, Steinley, (2004) provided the very simple formula $E({\rm RI})={n\choose 2}^{-2}\{(a+b)(a+c)+(c+d)(b+d)\}$, which entails that $E({\rm RD})={n\choose 2}^{-2}\{(a+b)(b+d)+(a+c)(c+d)\}$, so that

[TABLE]

This is very easy to implement, taking into account that $a=\sum_{i=1}^{r}\sum_{j=1}^{s}{n_{ij}\choose 2}$, $b=\sum_{j=1}^{s}{n_{+j}\choose 2}-a$, $c=\sum_{i=1}^{r}{n_{i+}\choose 2}-a$ and $d={n\choose 2}-a-b-c$ can be immediately computed from the confusion matrix (Jain and Dubes,, 1988, Section 4.4.1).

In contrast, computation of the MED requires solving a discrete minimization problem over $\max\{r!,s!\}$ possible inputs, so its implementation is not that simple, which surely hinders its usage. To fully describe the problem, assume that $r\leq s$ and define $n_{i+}=n_{ij}=0$ for all $i=r+1,\dots,s$ (if any). Writing $m_{ij}=n_{i+}+n_{+j}-2n_{ij}$ for $i,j=1,\dots,s$, then computation of the MED involves finding $\min_{\sigma\in\mathcal{P}_{s}}\sum_{i=1}^{s}m_{i,\sigma(i)}$, where $\mathcal{P}_{s}$ denotes the set of all possible permutations of $s$ elements. Despite its apparent complexity, this is a form of the well-known assignment problem, and very efficient algorithms exist to find its solution (see Burkard, Dell’Amico and Martello,, 2009). Appendix A below offers a simple implementation using the popular R language (R Core Team,, 2019).

3.2 Examples

To help understanding what the ARD and the MED represent and how they are computed in practice it is useful to start with some simple real data examples.

The first example regards the famous iris data set (Anderson,, 1935), including 4 measurements on $n=150$ flowers of three species of iris: Iris setosa, versicolor and virginica. If these data are clustered, e.g., using a normal mixture model (Fraley and Raftery,, 2002) with $G=3$ components, it results in the confusion matrix given in Table 1.

Thus, ${\rm MED}=(2+1)/150=0.02$ since only 3 data points would need to be re-labeled for the two partitions to coincide. On the other hand, there are $(2+1)\times(48+49)=291$ data pairs that belong to the same cluster in one of the partitions and to different clusters in the other, and that accounts for a proportion of ${\rm RD}=291/{150\choose 2}=0.026$ of the total number of possible data pairs. Finally, using Equation (5) the adjusted Rand distance for those two partitions is ${\rm ARD}=0.059$.

This is a very simple example because the two partitions have the same (small) number of clusters, and besides, they are quite similar. Nevertheless, it is helpful to perceive the differences between the MED, the RD and the ARD. Here, perhaps the MED is the easiest criterion to compute and to interpret, since it only involves counting misplaced individual data points. Obtaining the RD from the confusing matrix (by eye) is a bit more complex, since it implicates counting data pairs. And the corrected version ARD lacks the interpretability of the former two, but it still yields a very small number, indicating that the two partitions have a high degree of agreement.

Our second example concerns the DLBCL data set, introduced in Aghaeepour et al., (2013). It contains the records of the CD3, CD5 and CD19 antibodies on a set of $n=8183$ cells of a patient with Diffuse Large B-cell Lymphoma (DLBCL), along with the true cluster labels in five groups ($A$ to $E$) manually found by an expert. In Chacón, (2019), this data set was analyzed using several component merging techniques for mixture model clustering, in particular through the so-called modclust and entmerge methods. The former suggested the existence of three clusters, while the latter correctly identified five clusters; both confusion matrices are given together in Table 2.

Regarding the confusion matrix for the modclust labels, again it is not hard to compute the MED: the group matching leading to a higher degree of agreement would be $B$ with 2, $C$ with 3 and $E$ with 1, whereas the remaining $47+197+7+153+278+62+2=746$ data points would need to be re-labeled to make the two partitions coincide, thus yielding ${\rm MED}=746/8183=0.091$. Similarly, for the entmerge labels it can be checked that ${\rm MED}=1040/8183=0.127$, so that the modclust clustering is closer to the true expert labels regarding the MED, despite showing a smaller number of clusters. The reason is that, despite the entmerge method returned the true number of clusters, its assignments to clusters 4 and 5 were so unfortunate (especially, the splitting of cluster $B$ into two significant groups in clusters 3 and 4) that a high number of re-labelings is needed to make this partition equal to the true one. In contrast, the ARD for these two confusion matrices can be computed to be ${\rm ARD}=0.112$ and ${\rm ARD}=0.097$ for the modclust and entmerge partitions, respectively. As noted before, this does not yield such an intelligible comparison regarding the relative closeness of the two data-based partitions to the true clustering, because the baseline $E({\rm RD})$ is different for the two contingency tables. Nevertheless, it must be noted that the unadjusted distances ${\rm RD}=0.055$ and ${\rm RD}=0.047$ (respectively) also suggest that, in terms of data-pair disagreements, the entmerge clustering seems to be slightly closer to the expert partition than the modclust one.

The two previous examples illustrate the common scenario in real data analysis, where data-based partitions are not too dissimilar from the true clustering. To finish this section, a synthetic example concerning quite distant partitions is examined. The confusion matrix shown in Table 3 corresponds to the two assignments of $n=13$ objects into $r=s=5$ clusters in Table 2 in Steinley, (2003).

To appreciate how distant these two clusterings are, it is worth noting that ${\rm ARD}=1.164$, greater than 1, meaning that the disagreement between the two is higher than the average that would be obtained if the labels were randomly assigned (following the null model). The number of data pairs that are in the same group in one clustering and in different groups in the other can be computed to be 22, out of the total of ${13\choose 2}=78$ possible data pairs, which leads to ${\rm RD}=22/78=0.282$. And, by considering any permutation of the columns of the confusion matrix that preserves all its diagonal entries as 1, adding up the off-diagonal figures leads to ${\rm MED}=8/13=0.615$.

This example further illustrates how counting “discordant” data pairs seems to be less intuitive than counting “discordant” individual data points. But also, it shows that the permutation for which the MED is attained may not be unique: for instance, rearranging the columns of the confusion matrix according to the permutation $(45123)$ yields the same MED value, as already noted in Steinley, (2003, 2004). In any case, it is easy to check that the values of the RD and the ARD also remain the same under that permutation. Such a phenomenon is expected to occur for the comparison of very dissimilar clusterings; for example, in the extreme case where the confusion matrix ${\mathbf{N}}$ has all its entries equal to 1 (representing independent label assignments), then any permutation of its column leads to the same MED, RD and ARD values.

3.3 Two clusters in each clustering

In order to gain a deeper understanding of the behaviour of the MED and the ARD the next step is to analyze in detail the simplest scenarios. Arguably, the simplest comparison between two clusterings arises when either $r=1$ or $s=1$, but that could be considered a degenerate case, since in fact one of the partitions would show no clusters. So the next simplest case is $r=s=2$; we will focus our attention on this case first, and then we will generalize some of our findings to the case of arbitrary $r$ and $s$.

Independently of the criterion employed to compare clusterings, any researcher would probably agree that having a diagonal confusion matrix is synonymous with a perfect agreement between the two partitions. But that is also the case if the confusion matrix is anti-diagonal, which means, for $r=s=2$, that

[TABLE]

This clearly illustrates a key difference between classification and clustering: since classification is a supervised learning problem, the training data are already equipped with precise-meaning labels, and hence an anti-diagonal confusion matrix must be interpreted as the result of a totally wrong classification; in contrast, a clustering algorithm labels the groups as it finds them and, hence, the coding designation is not important (group 1 might as well have been called group 2, and viceversa) so that an anti-diagonal confusion matrix also represents perfect agreement, since the discovered groups are exactly the same, only differing in their (arbitrary) denomination. Mathematically, this means that distances between clusterings must be invariant with respect to permutations of the cluster labels (Meilă,, 2012).

It is precisely the way to measure deviations from the diagonal or anti-diagonal situation what gives rise to the different distances between clusterings. For the case $r=s=2$, let us consider the $2\times 2$ confusion matrix ${\mathbf{N}}=(n_{ij})$, and denote $d_{1}=n_{11}+n_{22}$ and $d_{2}=n_{12}+n_{21}$ the total sum of its diagonal and anti-diagonal entries, respectively. In this case, it is not hard to show that the MED and the RD can be simply expressed as

[TABLE]

To graphically appreciate the differences between the MED and the RD, and noting that $d_{2}=n-d_{1}$, Figure 1 shows the possible values of these criteria for $n=20$, as a function of $d_{1}$. The linear and quadratic appearances of the MED and the RD, respectively, are explained by the fact that they can be equivalently expressed as ${\rm MED}=1/2-|d_{1}/n-1/2|$ and ${\rm RD}={n\choose 2}^{-1}d_{1}(n-d_{1})$.

On the other hand, it is not possible to express the ARD as a function of $d_{1}$ and $d_{2}$ only. For a given value of $d_{1}$, there exist configurations of the confusion matrix that result in different ARD values. Figure 1 also shows all these possible ARD values for each given $d_{1}$ (marked with a tick over the whole possible range, that is indicated with a vertical line). This reveals a somehow erratic behaviour of the ARD in some cases, and inspecting such cases more closely allows to clarify how the ARD works. For instance, for $n=20$ consider the confusion matrices

[TABLE]

The two matrices have $d_{1}=16$, so that ${\rm MED}=0.2$ and ${\rm RD}=0.337$ for both ${\mathbf{N}}_{1}$ and ${\mathbf{N}}_{2}$. However, ${\rm ARD}=1.097$ for ${\mathbf{N}_{1}}$ whereas ${\rm ARD}=0.668$ for ${\mathbf{N}}_{2}$. In the first configuration, in both clusterings there is a big cluster with 18 elements and a relatively small one with only 2 elements; both clusterings agree on most of the elements in the big cluster, but show no agreement at all regarding the small cluster, since none of the data points has been simultaneously assigned to the small cluster in both clusterings. In the second configuration, the first clustering presents two quite balanced clusters, say $\mathscr{C}=\{C_{1},C_{2}\}$, of sizes 11 and 9 (respectively), while the second clustering has clusters of sizes $15$ and $5$, which can be obtained from $\mathscr{C}$ by transferring 4 elements from $C_{2}$ to $C_{1}$. The ARD seems to penalize the first configuration much more severely than the second one.

3.3.1 Worst-case scenario

The previous formulas for the MED and the RD in terms of $d_{1}$ and $d_{2}$ are also useful to analyze the worst-case scenario; i.e., the situation in which two given clusterings are as dissimilar as possible. If $n$ is even, then the maximum possible MED is $1/2$ and it is attained for $d_{1}=d_{2}=n/2$. Thus, it is worth remarking that even for the two most dissimilar possible clusterings the MED is not going to be higher than $0.5$ for the case of $r=s=2$. This could make a case against the use of the MED, since one would expect this distance to attain a maximum of 1 when comparing the most dissimilar clusterings. However, a moment of reflection reveals that this maximum of $0.5$ makes perfect sense in the context of clustering comparison, due to the aforementioned feature that any cluster label permutation should not affect distances between clusterings: having a proportion of label disagreements greater than $0.5$ would mean than exchanging the label denominations we would get a proportion smaller than $0.5$.

Nevertheless, it is helpful to keep the value of the maximum possible distance in mind at the time of judging how far two clusterings are: a MED of $0.4$ always has the same interpretation, but in relative terms it represents a worse result if the maximum possible MED is $0.5$ than if it is $0.95$. Hence, this suggests the introduction of a normalized MED, defined as ${\rm NMED}={\rm MED}/\max{\rm MED}$, to record how large the MED is with respect to its maximum possible value (given fixed values of $r$, $s$ and $n$). This should not replace the unnormalized MED, since they offer different information, but they should be given together. In the previous example, having ${\rm MED=0.4}$, ${\rm NMED}=0.8$ versus ${\rm MED=0.4}$, ${\rm NMED}=0.42$ indicates that the former situation is closer to the case of totally dissimilar clusterings than the latter one. Notice that this is a very different adjustment from the usual one, since it is not based on the expected value of the index under some null model; indeed, it does not rely on any choice of a null model.

The difficulty of such a normalization is that it is necessary to analyze which is the worst-case scenario for each index. Continuing with the $2\times 2$ table, it is not hard to check that $\max{\rm MED}=(n-1)/(2n)$ if $n$ is odd, which is attained for both $d_{1}=(n-1)/2$ and $d_{1}=(n+1)/2$. Therefore, ${\rm NMED}=2n^{-1}\min\{d_{1},d_{2}\}$ for even $n$ and ${\rm NMED}=2(n-1)^{-1}\min\{d_{1},d_{2}\}$ for odd $n$.

Regarding the RD, its maximum is attained at the same value of $d_{1}$ as for the MED, resulting in $\max{\rm RD}=n/\{2(n-1)\}$ for even $n$ and $\max{\rm RD}=(n+1)/(2n)$ for odd $n$, so that it approaches $1/2$ as $n$ increases. Hence, the normalized RD, defined as ${\rm NRD}={\rm RD}/\max{\rm RD}$, can be explicitly written as ${\rm NRD}=4n^{-2}d_{1}d_{2}$ for even $n$ and ${\rm NRD}=4\{(n-1)(n+1)\}^{-1}d_{1}d_{2}$ for odd $n$.

For the ARD, it would have been expected that its maximum were attained amongst the possible configurations with $d_{1}=n/2$ for even $n$ (or $d_{1}=(n-1)/2$ for odd $n$), but Figure 1 shows that this does not happen, in general. For instance, for $n=20$ the maximum ARD is attained for a configuration with $d_{1}=12$; more precisely, for ${\mathbf{N}}=\left(\begin{smallmatrix}12&4\\ 4&0\end{smallmatrix}\right)$, which gives $\max{\rm ARD}=95/84\simeq 1.131$. It is somehow counterintuitive that the maximum value of the ARD is not attained for ${\mathbf{N}}=\left(\begin{smallmatrix}5&5\\ 5&5\end{smallmatrix}\right)$, which represents the situation where the labels of the first clustering are perfectly independent from the labels in the second clustering.

In fact, it would be interesting to study which are the possible maximum and minimum values of the ARD for a given $d_{1}$. Since $d_{1}$ and $d_{2}=n-d_{1}$ are fixed, the numerator in the definition of the ARD is constant, so this problem is equivalent to finding the minimum and maximum values of $E({\rm RD})$ for a given $d_{1}$. It appears (although it was not possible to find a simple proof) that for a given $d_{1}\geq d_{2}$, the maximum ARD is attained for

[TABLE]

provided $d_{2}\geq 2$ is even or odd, respectively, and that for $d_{1}=n-1,\,d_{2}=1$ the confusion matrix configuration that maximizes the ARD is $\left(\begin{smallmatrix}n-2&0\\ 1&1\end{smallmatrix}\right)$. Using the form for even $d_{2}$, the resulting maximum ARD for a given $d_{1}$ can be expressed as

[TABLE]

for $d_{1}\geq d_{2}\geq 2$. Maximizing $\alpha_{n}(d_{1})$ with respect to $d_{1}$ yields $\max{\rm ARD}$, but it is not clear how to obtain an explicit expression for such a maximum. In any case, note that it is not necessary to normalize the ARD, since this distance already includes a kind of normalization (although by the expected value of the RD, not by its maximum).

3.3.2 Close clusterings

Similarly, this in-depth inspection of the $2\times 2$ case is also beneficial to understand how these measures of dissimilarity between two clusterings evolve when such clusterings are very close. All these distances obviously return a zero value if $n_{12}=n_{21}=0$, but the question that will be addressed here is how these distances behave as $n_{12}\to 0$ and $n_{21}\to 0$ before reaching their null limit. More precisely, the goal is to provide a linear approximation of the MED and the RD for small values of $n_{12}$ and $n_{21}$.

Such an approximation is very easy to find for the MED, since as both $n_{12},n_{21}\to 0$ it is clear that $\min\{d_{1},d_{2}\}=n_{12}+n_{21}$, so that ${\rm MED}=(n_{12}+n_{21})/n$ for small values of $n_{12}$ and $n_{21}$ (this is an equality rather than an approximation). On the other hand, it is possible to write {\rm RD}={n\choose 2}^{-1}\big{\{}n(n_{12}+n_{21})-(n_{12}+n_{21})^{2}\big{\}}, so that a Taylor expansion gives ${\rm RD}\simeq 2(n_{12}+n_{21})/(n-1)$ as $n_{12},n_{21}\to 0$. This means that, for small values of $n_{12}$ and $n_{21}$, the RD will be roughly twice the MED.

For instance, for ${\mathbf{N}}=\left(\begin{smallmatrix}55&6\\ 4&35\end{smallmatrix}\right)$ we have ${\rm MED}=0.1$ and ${\rm RD}=0.182$, while the approximation formula for the RD reads $2\cdot(6+4)/(100-1)=0.202$.

3.4 Arbitrary number of clusters

The case $r=s=2$ is surely the easiest one to analyze in detail, and its analysis results in a deeper understanding of how the MED, the RD and the ARD behave. Here, such an analysis is extended for the comparison of two clusterings with an arbitrary number of clusters.

One of the findings for $r=s=2$ is that the MED attains its maximum when the clustering labels are perfectly independent. In general, this refers to the situation where $n$ is a multiple of $rs$ and the $(r\times s)$-confusion matrix ${\mathbf{N}}$ has all its entries equal to $n/(rs)$. In that case, note that $\sum_{i=1}^{r}n_{i,\sigma(i)}=n/s$ for any $\sigma\in\mathcal{P}_{s}$ and, therefore, ${\rm MED}=1-1/s$. But, assuming $r\leq s\leq n$, Charon et al., (2006, Lemma 1) showed that an upper bound for the MED is $1-\lceil n/s\rceil/n$ (with $\lceil\cdot\rceil$ standing for the ceiling function), which generalizes the bounds obtained for odd or even $n$ in the previous section for $s=2$. Hence, once again the maximum MED is attained for the case of perfectly independent clustering labels. Thus, the corresponding normalized MED is defined in general as ${\rm NMED}=\frac{n}{n-\lceil n/q\rceil}{\rm MED}$, where $q=\max\{r,s\}$. The effect of this normalization is noticeable for small values of $q$, but becomes negligible as $q$ increases.

In contrast, the story for the RD with arbitrary $r\leq s$ is different from the case $r=s=2$. An exhaustive enumeration of all the possible confusion matrix configurations for small values of $r$, $s$ and $n$ suggests that, given $r\leq s$ and $n\geq 2(r-1)+s$, the maximum value of the RD is always attained for a matrix of the form

[TABLE]

with $q_{1},\dots,q_{s}\in\mathbb{N}$ and $q_{1}\geq\dots\geq q_{s}$ (it remains open to find a formal proof of this fact). This does not mean that the maximizing matrix is necessarily unique; in fact, as noted before, for $r=s=2$ and $n=20$ the maximum RD is attained for any confusion matrix with $d_{1}=10$, for instance for ${\mathbf{N}}=\left(\begin{smallmatrix}10&9\\ 1&0\end{smallmatrix}\right)$ which has the form (7), and also for ${\mathbf{N}}=\left(\begin{smallmatrix}5&5\\ 5&5\end{smallmatrix}\right)$, which represents the perfectly independent situation. In addition, assuming that the conjectured form of the maximizer is correct, it is shown in Appendix B that $\max{\rm RD}$ is attained by taking $q_{1}=k+r-1$, the next $\ell\geq 0$ coordinates $q_{2}=\cdots=q_{\ell+1}=k+1$ and the remaining $s-\ell-1\geq 0$ coordinates $q_{\ell+2}=\cdots=q_{s}=k$, where $k=\lfloor\{n-2(r-1)\}/s\rfloor\in\mathbb{N}$ (with $\lfloor\cdot\rfloor$ standing for the floor function) and $\ell=n-2(r-1)-ks\in\{0,1,\dots,s-1\}$; that is, $k$ and $\ell$ are the quotient and the remainder of the (Euclidean) division of $n-2(r-1)$ by $s$, respectively. With such a choice, it follows that

[TABLE]

This allows to explicitly define the normalization ${\rm NRD}={\rm RD}/\max{\rm RD}$.

Further, when $n$ is a multiple of $rs$ and the two clusterings are perfectly independent it is easy to check that $n(n-1){\rm RD}=n^{2}(r+s-2)/(rs)$. Since $\max{\rm RD}\sim 1-1/s$ as $n\to\infty$, it follows that the maximum ${\rm RD}$ is not attained for perfectly independent clusterings for big enough $n$ if $s>2$. Moreover, in practice this seems to be the case for all $n$, as shown in Figure 2. This figure represents the normalized RD for the case of two perfectly independent clusterings for several combinations of $r$ and $s$. Only for the pair $(r,s)=(2,2)$ the RD for perfectly independent clusterings matches its maximum possible value. For any other combination, having two totally unrelated clusterings does not yield the maximum possible RD; indeed, this phenomenon becomes more and more severe as $r+s$ increases and, for instance, for $(r,s)=(5,5)$ and $n=100$ the confusion matrix with all its entries equal to 4 results in a RD that is only $42\%$ of the maximum achievable RD, attained for a matrix of the form (7) with $q_{1}=22$, $q_{2}=q_{3}=19$ and $q_{4}=q_{5}=18$.

Finally, it is worth noting that $(r+s-2)/(rs)$ decreases as $r$ and/or $s$ increases, so that the RD for the case of perfectly independent clusterings becomes quite small when both $r$ and $s$ are large and, hence, the RD does not seem useful to detect this important instance of unrelated clusterings. Fowlkes and Mallows, (1983, p. 555) already noted this phenomenon, upon inspecting the expected value and variance of the Rand index under the null model.

For the ARD, it was not possible to provide an explicit formula for its maximum for $r=s=2$, and the problem is of course more intricate for arbitrary $r$ and $s$. Nevertheless, it seems clear that the maximum ARD is not attained for the case of independent clustering labels, in general. Instead, the inspection of all possible confusion matrix configurations for small values of $r$, $s$ and sufficiently large $n$ seems to suggest that the maximum value of the ARD is always attained for a matrix of the form

[TABLE]

with $p_{1},\dots,p_{r},q_{2},\dots,q_{s}\in\mathbb{N}$ and $p_{1}\geq p_{2}\geq\cdots\geq p_{r}$, $p_{1}\geq q_{2}\geq\cdots q_{s}$ (furthermore, with $(p_{2},\dots,p_{r})=(q_{2},\dots,q_{s})$ if $r=s$). Indeed, confusion matrices with ARD larger than the value corresponding to the perfectly independent case can be constructed by following the guidelines described above for $r=s=2$. For instance, if $n=24$, $r=2$, $s=3$, then the confusion matrices

[TABLE]

lead to ARDs of $1.062$ and $1.143$, respectively, and for $n=27$, $r=3$, $s=3$, the confusion matrices

[TABLE]

yield ARDs of $1.083$ and $1.157$, respectively. Moreover, when $n$ is a multiple of $rs$ and the two clusterings are perfectly independent, it is easy to show that ${\rm ARD}=(n-1)/\{n-(2rs-r-s)/(r+s-2)\}$, which approaches 1 (from above) as $n$ increases.

4 Numerical experiments

In this section, the distributions of the MED, the RD, the ARD and the normalized versions NMED and NRD will be compared in different simulated scenarios.

As noted in Van Mechelen et al., (2018), benchmarking studies for cluster analysis do not abound. Nevertheless, the task of comparing different external criteria via simulation was addressed in the seminal paper by Milligan and Cooper, (1986) and also more recently in Steinley, (2004), Denœud and Guénoche, (2006) or Steinley and Brusco, (2018).

Broadly speaking, these studies handle two possible scenarios. The first one explores the performance of the criteria in the null case, that is, when the agreement between the compared clusterings is only due to chance. And the second framework concerns how the criteria of interest behave as the two compared clusterings drift apart, starting from perfect similarity. Both scenarios are considered separately in the next sections.

4.1 The null case

As noted above, the null case scenario covers the situation where the clustering agreements are solely due to chance. However, as remarked in Gates and Ahn, (2017), different choices for the model for random clusterings can be made, and a careful model selection is needed to provide a baseline that is neither based on a model that “is not random enough” nor on a model that is “too random”.

These authors considered three models for random clusterings, with increasing level of randomness, starting with the permutation model (where the number of clusters and their sizes is fixed), followed by the model where only the number of clusters is fixed, and finally the model encompassing all possible clusterings, with arbitrary number of clusters and cluster sizes. As a compromise for intermediate randomness level, in this section the null case refers to the situation where random labels are drawn uniformly after fixing the number of clusters.

Hence, the distribution of the considered criteria in the null case is explored by computing their values on a big enough number $B$ of random clustering pairs of $n$ objects, obtained by independently drawing two uniform samples of size $n$ from $\{1,\dots,r\}$ and $\{1,\dots,s\}$, respectively. The number of synthetic replicates was set to $B=10000$, in order to obtain a precise approximation of the distributions; the number of clusters was considered equal ($r=s$), in common with some of the aforementioned previous studies, and ranging in $\{2,3,4,5\}$; and samples sizes $n=100$ and $n=400$ were used to investigate the effect of an increasing number of data points. The distributions of the studied criteria are depicted in Figures 3 and 4 for $n=100$ and $n=400$, respectively, by means of side-by-side vertical histograms (with the bars mirrored with respect to the vertical axis), whose bars have been rescaled so that each of them has maximum bar length equal to 1, to aid visualization.

Despite being corrected for chance according to the permutation model (which is not exactly the null model in this study), the distribution of the ARD seems to be centered at 1 in all cases, so this type of adjustment makes it possible to compare its behaviour along the different configurations. Its variability is the second lowest among the compared criteria, it seems not to change with the number of clusters but quickly decreases with the sample size, as also noted in Steinley, Brusco and Hubert, (2016). This suggests that the ARD may give rise to a powerful tool for detecting clustering independence.

The RD is the least variable criterion out of those considered here. This is not surprising in view of the previous Figure 1, since its quadratic nature entails a least pronounced descent around its null-case value than the MED, for instance. Besides, as remarked in the previous section, under this null scenario the RD only achieves its maximum value for the case $r=s=2$, which yields a tightly concentrated distribution of the NRD with a maximum of 1 in that case. However, as previously shown in Figure 2, the maximum possible value of the RD becomes quite bigger than its value for independent clusterings as the number of clusters increases, and this explains why even the distributions of the normalized RD are far from 1. In other words, confusion matrices corresponding to randomly generated clusterings are usually far from something like (7). Probably, it might be possible to obtain NRD distributions much closer to 1 if the random clusters were generated to produce confusion matrices only slightly deviated from (7), but that does not seem to be an appropriate null model.

The MED is notably more variable than the ARD and the RD, with standard deviations about 1.6–2.1 times greater than those of the ARD, and 3.5–4.2 times greater than those of the RD, for $n=100$ (3.1–4.2 and 6.7–8.5, respectively, for $n=400$). Its variability, though, appears to decrease slightly as the number of clusters grow. Its approximated distribution shows an upper bound that agrees with the results in the previous section (for $n=100$, e.g., a maximum value of 0.5, 0.66, 0.75 and 0.8 for $r=s=2,3,4,5$, respectively), yielding location features that naturally change with the number of clusters, and hence making inappropriate to aggregate its results across the different simulated configurations. This upper bound also implies that NMED certainly attains a maximum value of 1 for this null scenario of random clusterings. However, it must be pointed out that the probability of attaining such a maximum value seems to decrease with the number of clusters.

Indeed, in some cases it is possible even to give an exact expression for such a probability. For $r=s=2$ and even $n$, for instance, it corresponds to $P(d_{1}=n/2)$, where $d_{1}$ is the sum of the two diagonal terms in the confusion matrix. In the null scenario, $d_{1}$ is a random variable following a binomial distribution, with $n$ as the number of trials and probability of success $p=1/2$ (the probability that two uniform and independent choices from $\{1,2\}$ are the same). Hence, P(d_{1}=n/2)={n\choose n/2}\big{/}2^{n}. More generally, here the random variable $n\cdot{\rm MED}$ follows a folded binomial distribution (Gart,, 1970).

4.2 Diverging clusterings

The second simulation scenario concerns studying the evolution of the compared criteria as two clusterings move away from each other, starting from a situation of perfect agreement.

Interestingly, in most of the existing simulation studies (see, for instance, Steinley,, 2004, Denœud and Guénoche,, 2006), the process of “moving away from each other” is quantified by measuring the proportion of data points that are differently clustered from the initial stage of perfect agreement. Steinley, (2004) called this proportion the “degree of overlap”, and more recently Steinley and Brusco, (2018, Section 3.2.2) referred to this measure of deviation from the perfect agreement as the misclassification rate. So, overall, this simulation scenario concerns inspecting how the other clustering distances evolve as compared to the MED (see Figure 3 in Steinley,, 2004, or Figure 1 in Denœud and Guénoche,, 2006).

Aside, it should be noted that what Steinley, (2004) and Steinley and Brusco, (2018) called degree of overlap is not exactly the same as the MED. The simulation setup in these references concerns a diagonal confusion matrix as the starting point (hence, a situation of perfect agreement) that is progressively perturbed by randomly taking a proportion of objects from the diagonal and placing them in off-diagonal cells. This proportion of off-diagonal objects is what is called degree of overlap (DO), and in the aforementioned studies it is allowed to vary in $0.05,0.10,\dots,0.95$. However, this is not the same as the misclassification rate: while the DO and the MED usually coincide for low DO values, when the DO is too high it may occur that the resulting clusterings became indeed closer with respect to the MED, instead of further away. For example, consider the confusion matrices

[TABLE]

For all of them, $n=20$. Matrix ${\mathbf{N}}_{2}$ stems from ${\mathbf{N}}_{1}$ after removing a total of 13 objects from the diagonal, so that ${\rm DO}=13/20=0.65$ for ${\mathbf{N}}_{2}$; it can be checked that ${\rm MED}=0.65$ for ${\mathbf{N}}_{2}$ as well. Five additional objects are removed from the diagonal when going from ${\mathbf{N}}_{2}$ to ${\mathbf{N}}_{3}$, representing a total ${\rm DO}=18/20=0.9$ with respect to ${\mathbf{N}}_{1}$, but ${\rm MED}=0.4$ for ${\mathbf{N}}_{3}$, lower than for ${\mathbf{N}}_{2}$. Of course, this is due to the fact that $\max{\rm MED}=0.65$ for $n=20$ and $r=s=3$, so that it does not seem appropriate to consider DO values greater than $0.65$ in this case.

In Denœud and Guénoche, (2006), several agreement indices were compared as a function of an increasing MED. A given starting partition is recursively perturbed by randomly selecting one element and a new class label for it. This procedure aims at randomly and equiprobably generating partitions at precise MED of the given one. However, there the number of clusters is not fixed and, hence, their study comprises a higher degree of uncertainty.

Nevertheless, for the goal of inspecting the evolution of the different distances as a function of the MED, the most exhaustive procedure is surely that based on computing the involved measures for all the possible confusion matrix configurations, that is, for all the matrices in $\mathcal{N}(r,s,n)=\{{\mathbf{N}}\in\mathcal{M}_{r\times s}\colon n_{i+}>0$ for all $i$, $n_{+j}>0$ for all $j$, and $\sum_{i,j}n_{ij}=n\}$. Indeed, this is precisely what Figure 1 represents for $r=s=2$ and $n=20$. But this could be accomplished in that case because the cardinality $|\mathcal{N}(2,2,20)|=1691$ was reasonably small.

For $r=s=3$ and $n=20$ the class of all possible confusion matrices is considerably larger, namely $|\mathcal{N}(3,3,20)|=2806281$, but still not prohibitive, so its exhaustive enumeration is yet feasible. Therefore, the ARD, the MED and the RD were obtained for each of these possible confusion matrix configurations, yielding a large amount of interesting information. Figure 5 shows boxplots for the conditional distributions of the RD (left) and the ARD (right), given the MED, along with the (mean) regression curve. These plots contain the same information as Figure 1, but a first notable difference is that now the RD corresponding to a given MED is no longer a single value, as it happened for $r=s=2$; instead, for $r=s=3$ all the possible confusion matrices with the same MED result in a wide range of different RD values.

Most of the conditional distributions given the MED are fairly symmetric, but it is worth remarking some interesting features that arise, especially, for very low or very high MED values. For instance, Figure 6 focuses on the distribution of the RD given the particular values of ${\rm MED}=0.1$ (left) and ${\rm MED}=0.6$ (right), which clearly show a high degree of skewness. The conditional distribution of the ARD also shows some peculiarities: its maximum value (${\rm ARD}=1.205$) is attained for a confusion matrix with ${\rm MED}=0.5$, but its conditional mean attains its maximum at ${\rm MED}=0.65$ (the maximum possible MED value). The outlier in the conditional distribution of the ARD given ${\rm MED}=0.2$ is particularly striking, with a value of $1.108$, attained at the confusion matrix

[TABLE]

despite it is only at 4 data point transfers from the perfect agreement situation.

This extensive enumeration study is also useful to inspect the individual distributions of each criterion. For example, Figure 7 shows the distribution of all the MED values for $r=s=3$ and $n=20$, and reveals a very different scenario from the one that Figure 1 in Steinley, (2004) suggested. In Steinley’s simulation study, the distribution of the MED appeared to be somewhat uniform, which strongly contrasts with the distribution shape shown in Figure 7 from exhaustive enumeration. The reason is that in Steinley’s paper the distribution of the MED is investigated by aggregation of all the multiple simulation conditions. And, as noted before, these simulation conditions involved uniformly varying the DO from $0.05$ to $0.95$. It was already noted above that the DO is not exactly the same as the MED, but they are closely related, so forcing a fixed given number of simulations for every DO level naturally results in a (nearly) uniform distribution for the MED. However, the exhaustive inspection of all the possible confusion matrix configurations in Figure 7 shows that the MED distribution is quite different from the uniform one.

For higher values of $r$, $s$ or $n$, it is not possible to enumerate all the confusion matrices in $\mathcal{N}(r,s,n)$ anymore, since its cardinality becomes exorbitant. An alternative way to approximate the criteria distributions, for these higher values of $r$, $s$ and $n$, would be to randomly sample a large number of matrices from $\mathcal{N}(r,s,n)$ (in an equiprobable way), and then compute their MEDs, RDs and ARDs in order to obtain an equivalent approximation of Figure 5. This suggestion is not without problems, either, for two reasons: first, it is not straightforward to uniformly sample from $\mathcal{N}(r,s,n)$, see Appendix C for a valid procedure; and second, the fact that some of the possible MED values occur only for a low number of confusion matrices makes it difficult to procure an accurate approximation of the conditional distributions given such MED values. For example, from the exhaustive enumeration of $\mathcal{N}(3,3,20)$ it follows that the probability of obtaining a confusion matrix with ${\rm MED}=0.65$ by uniform sampling is approximately $1.6\times 10^{-3}$, so a large simulation size would be required in order to approximate the distribution of the other distances given ${\rm MED}=0.65$.

In any case, following the procedure suggested in Appendix C, a random sample of size $10^{7}$ was drawn from $\mathcal{N}(5,5,80)$, and the values of the MED, RD and ARD for these confusion matrices were recorded. It must be remarked that the cardinality of $\mathcal{N}(5,5,80)$ is approximately $2.309\times 10^{23}$, which makes the exhaustive enumeration approach unfeasible. From that sample, it is possible to approximate the conditional means of the RD and ARD given the MED (Figure 8, left) and to provide an approximate analogue of Figure 7 for $n=80$ and $r=s=5$ (Figure 8, right). Notice also that, even if the possible MED values are $\{i/80\colon i=0,1,\dots,64\}$ (because $\max{\rm MED}=0.8$ for $r=s=5$ and $n=80$), the range of MED values for which 10 or more observations were obtained in this particular sample reduced to $\{i/80\colon i=15,16,\dots,60\}$ (i.e., the others had sample frequencies smaller than $10^{-6}$), and that is why Figure 8 presents some missing parts.

5 Discussion

Regarding the task of comparing two partitions of a finite data set, surely the confusion matrix is the object that yields the most complete information. However, when it has a considerable number of cells, it provides somehow too many details and it becomes necessary to resort to some summary statistic to extract useful information. The MED, the RD and the ARD are examples of such summary statistics, each of them offering a different synopsis.

The first two represent empirical versions of distances between whole-space clusterings and, intuitively, correspond to computing the proportion of “differently placed” individual data points (in case of the MED) or data pairs (for the RD) along the compared clusterings. Considering data pairs instead of individual data points appears somehow less intuitive. In fact, as pointed out in Hubert and Arabie, (1985, Section 4), one could equally consider using data triplets or, more generally, data $k$-tuples. Comparisons, however, become more and more intricate as $k$ increases. It is in this sense that the choice of $k=1$ (i.e., the MED) represents the simplest option.

But it is not just a matter of simplicity. As long as $\min\{r,s\}>2$, the RD also shows a more serious and undesirable drawback: the case of completely unrelated clusterings does not correspond to the most dissimilar clustering pair, according to the RD, and this phenomenon becomes more and more severe as the clustering sizes increase (as shown in Figure 2). This unfortunate feature is not shared by the MED, which does point out unrelated clusterings as an instance of extreme dissimilarity, and furthermore shows a maximum value that quickly approaches 1 as $\max\{r,s\}$ increases.

A possibility to correct the aforementioned flaw is to consider the relative size of the RD with respect to the average RD value when the two clusterings are generated at random, this is what the ARD provides. This exhibits the natural advantage of creating a criterion that is always centred at 1 for the null case, but on the other hand introduces a distorting element that further complicates the interpretation: now the ARD represents the relative size of the proportion of differently treated data pairs in the compared clusterings with respect to a baseline, taken as the average value of that proportion when the cluster labels are assigned at random while maintaining the number of clusters and cluster sizes fixed. This also implies that, if the baseline changes (as usually happens when inspecting two different confusion matrices), then the relative comparison of the two scenarios by means of ARD scores becomes unclear.

An alternative remedy, also aimed to examine the relative size of a criterion, but this time against the worst possible case, is to normalize such a criterion with respect to its maximum value. This is a different kind of adjustment, which does not produce a criterion that is centred at 1 for a null model (in fact, it does not rely on a specific null model), however it ensures that all the resulting values lie on $[0,1]$ instead. When applied to the MED and the RD it results in the new NMED and NRD criteria, which are not advised to be used alone, but jointly with their unnormalized counterparts, since the latter retain the most straightforward interpretation. In addition, not achieving its maximum for unrelated clusterings also hinders this approach for the RD, as shown in Figures 3 and 4, since it entails that the distribution of the NRD can be far from 1 under the null model. In contrast, the NMED distribution is indeed close to its upper bound of 1 in the null case, more so for higher sample sizes, although it must be pointed out that it seems more and more unlikely to reach this upper bound as the number of clusters increase.

In any case, it seems clear that the study of the distributions of all these criteria (the MED, the RD and the ARD) deserves further consideration, since its investigation through exhaustive enumeration or uniform sampling from the set of all possible confusion matrices has revealed some previous misconceptions and unexpected features.

Acknowledgments. The author acknowledges the support of the Spanish Ministerio de Economía y Competitividad grant MTM2016-78751-P and the Junta de Extremadura grant GR18016.

Appendix A: R function for misclassification error distance computation

Recall from Section 3.1 that, given a confusion matrix ${\mathbf{N}}=(n_{ij})\in\mathcal{N}(r,s,n)$ with $r\leq s$, the main computational problem is to find

[TABLE]

where $\mathcal{P}_{s}$ denotes the set of all possible permutations of $s$ elements. Here, $n_{i+}=n_{ij}=0$ for all $i=r+1,\dots,s$ (if any) and $m_{ij}=n_{i+}+n_{+j}-2n_{ij}$ for $i,j=1,\dots,s$. Fortunately, (9) is a linear sum assignment problem, whose solution can be efficiently found through the function solve_LSAP included in the R library clue (Hornik,, 2005, 2018). So once that library is loaded, with the command library(clue), a function to compute the MED from two equal-size vectors containing the cluster labels with respect to each clustering can be obtained through the following simple code:

med <- function(labels1, labels2){ n <- length(labels1) N <- table(labels1, labels2) r <- nrow(N) s <- ncol(N) if (r>s) N <- t(N); r <- nrow(N); s <- ncol(N) if (r<s) N <- rbind(N, matrix(0, nrow = s-r, ncol = s)) M <- matrix(rowSums(N), nrow = s, ncol = s) + matrix(colSums(N), nrow = s, ncol = s, byrow = TRUE) - 2 * N optimal.permutation <- solve_LSAP(M) result <- sum(M[cbind(seq_along(optimal.permutation), optimal.permutation)]) / (2 * n) return(result) }

Appendix B: The maximum Rand distance

Assuming as true the conjecture that there is always a maximizer of the RD of the form (7), here it is shown that the maximum value of the RD satisfies (8). First notice that, for a confusion matrix of the form (7), the function $M(q_{1},\dots,q_{s})=n(n-1){\rm RD}=n^{2}\widehat{d}_{\rm H}$ can be explicitly written as

[TABLE]

where $c=(n-r+1)^{2}+(r-1)(r-2)$. Then, the goal is to maximize $M(q_{1},\dots,q_{s})$ with the constraint that the total number of data points is $n$, which for the matrix (7) yields $\sum_{j=1}^{s}q_{j}+r-1=n$. The method of Lagrange multipliers yields the maximizer over real-valued choices of $q_{1},\dots,q_{s}$ as $q_{1}^{*}=\{n-2(r-1)\}/s+r-1$, $q_{2}^{*}=\cdots=q_{s}^{*}=\{n-2(r-1)\}/s$, but recall that the goal is to find the maximizer for nonnegative integer values of $q_{1},\dots,q_{s}$.

As in Section 3.4, write $n-2(r-1)=ks+\ell$, with $k\in\mathbb{N}$ and $\ell\in\{0,1,\dots,s-1\}$. If $\{n-2(r-1)\}/s\in\mathbb{N}$ (corresponding to the case $\ell=0$), then the real-valued maximizer is also integer-valued and leads to the maximizer and maximum value announced in (7) and (8) for $\ell=0$.

If $\ell>0$ then the real-valued maximizer is rational, with all the coordinates $q_{1}^{*},\dots,q_{s}^{*}$ having the same fractional part $\ell/s$. Due to the total sum constraint, to find the integer-valued maximizer of (10) these fractional reminders need to be re-distributed into the coordinates $q_{1},\dots,q_{s}$, to make them integer, while at the same time trying to decrease the value of $M(q_{1},\dots,q_{s})$ as less as possible with respect to the real-valued maximizer. To achieve this, first note that (10) is a concave function with the same curvature along every direction, so the least decrease with integer coordinates with respect to the real-valued maximum corresponds to rounding up to the least greater integer as few coordinates of $q_{1}^{*},\dots,q_{s}^{*}$ as possible. Having $s$ fractional reminders of size $\ell/s$, that entails that the integer-valued maximizer is found by rounding up exactly $\ell$ coordinates to the least greater integer (and rounding down the remaining $s-\ell$ coordinates). Finally, since $q_{2},\dots,q_{s}$ play a symmetric role in (10), the only two cases to study comprise, either to round up $q_{1}^{*}$ and $\ell-1$ of the remaining coordinates, or to round up $\ell$ coordinates among $q_{2}^{*},\dots,q_{s}^{*}$ (say, the first $\ell$ of them). The first of these cases yields $q_{1}=k+r$, $q_{2}=\cdots=q_{\ell}=k+1$, $q_{\ell+1}=\dots=q_{s}=k$, while the second one entails $q_{1}=k+r-1$, $q_{2}=\cdots=q_{\ell+1}=k+1$, $q_{\ell+2}=\dots=q_{s}=k$. It is easy to check that these two choices achieve the same value for $M(q_{1},\dots,q_{s})$, and the second one agrees with the form posited in (7) and (8), which is, thus, valid for arbitrary $\ell$.

Appendix C: Uniform sampling from $\mathcal{N}(r,s,n)$

A composition of a positive integer $n$ into $k$ parts is a representation $n=\sum_{i=1}^{k}p_{i}$ in which $p_{1},\dots,p_{k}$ are non-negative integers and the order of the summands matters. There are $J(n,k)={n+k-1\choose n}$ possible compositions of $n$ into $k$ parts, and it is easy to draw a composition at random (uniformly) without necessarily generating the set of all of them (see Nijenhuis and Wilf,, 1978, Chapters 5 and 6).

The entries of any confusion matrix ${\mathbf{N}}\in\mathcal{N}(r,s,n)$ constitute a composition of $n$ into $r\cdot s$ parts. The additional conditions $n_{i+}>0$ and $n_{+j}>0$ for all $i,j$, imposed in the definition of $\mathcal{N}(r,s,n)$ to ensure that the sizes of the associated compared clusterings match $r$ and $s$, respectively, can be checked after arranging each drawn composition of $n$ into $r\cdot s$ parts by columns (say) into an $r\times s$ matrix, so uniform sampling from $\mathcal{N}(r,s,n)$ is guaranteed by rejection sampling.

For the simulations in Section 4.2 a sample of size $10^{7}$ was drawn from $N(5,5,80)$ using the previous approach. The recorded rejection rate during the process was very low, approximately $0.47\%$, so the sampling algorithm is very efficient. Moreover, that rejection rate can also be interpreted as an estimate of the proportion of compositions of $n=80$ into $r\cdot s=25$ parts that cannot be converted (by columns) into a confusion matrix of $\mathcal{N}(5,5,80)$ and, since $J(80,25)\simeq 2.319\times 10^{23}$, that yields an estimate of $2.309\times 10^{23}$ for the cardinality of $\mathcal{N}(5,5,80)$.