Comparing high dimensional partitions, with the Coclustering Adjusted Rand Index

TL;DR

This paper introduces the Co-clustering Adjusted Rand Index (CARI) for measuring agreement between co-clustering partitions, along with improvements to existing criteria, validated through experiments on simulated and real data.

Contribution

It proposes a new co-clustering agreement measure, CARI, and enhances existing criteria like Classification Error and Normalized Mutual Information for better co-clustering evaluation.

Findings

CARI effectively measures co-clustering agreement.

Improved criteria show better properties in experiments.

Experimental results validate the proposed measures.

Abstract

We consider the simultaneous clustering of rows and columns of a matrix and more particularly the ability to measure the agreement between two co-clustering partitions. The new criterion we developed is based on the Adjusted Rand Index and is called the Co-clustering Adjusted Rand Index named CARI. We also suggest new improvements to existing criteria such as the Classification Error which counts the proportion of misclassified cells and the Extended Normalized Mutual Information criterion which is a generalization of the criterion based on mutual information in the case of classic classifications. We study these criteria with regard to some desired properties deriving from the co-clustering context. Experiments on simulated and real observed data are proposed to compare the behavior of these criteria.

Tables

Table 1. Table 1: Contingency table to compare two pairs of co-clustering partitions.

Table 2. Table 2: Initial contingency table 𝒏 𝒛 ​ 𝒘 ​ 𝒛 ′ ​ 𝒘 ′ superscript 𝒏 𝒛 𝒘 superscript 𝒛 bold-′ superscript 𝒘 bold-′ \boldsymbol{n^{zwz^{\prime}w^{\prime}}} (see Definition 2.1).

Block	$(1,1)$	$(1,2)$	$(1,3)$	$(1,4)$	$(2,1)$	$(2,2)$	$(2,3)$	$(2,4)$	$(3,1)$	$(3,2)$	$(3,3)$	$(3,4)$
$(1,1)$	0	0	0	0	0	4	0	0	0	0	0	0
$(1,2)$	0	0	0	0	2	0	0	0	0	0	0	0
$(1,3)$	0	0	0	0	0	0	0	2	0	0	0	0
$(1,4)$	0	0	0	0	0	0	2	0	0	0	0	0
$(2,1)$	0	0	0	0	0	0	0	0	0	2	0	0
$(2,2)$	0	0	0	0	0	0	0	0	1	0	0	0
$(2,3)$	0	0	0	0	0	0	0	0	0	0	0	1
$(2,4)$	0	0	0	0	0	0	0	0	0	0	1	0
$(3,1)$	0	2	0	0	0	0	0	0	0	0	0	0
$(3,2)$	1	0	0	0	0	0	0	0	0	0	0	0
$(3,3)$	0	0	0	1	0	0	0	0	0	0	0	0
$(3,4)$	0	0	1	0	0	0	0	0	0	0	0	0

Table 3. Table 3: Contingency tables 𝒏 𝒛 ​ 𝒛 ′ superscript 𝒏 𝒛 superscript 𝒛 bold-′ \boldsymbol{n^{zz^{\prime}}} (on the left) and 𝒏 𝒘 ​ 𝒘 ′ superscript 𝒏 𝒘 superscript 𝒘 bold-′ \boldsymbol{n^{ww^{\prime}}} (on the right).

Cluster	1	2	Margin
1	2	0	2
2	2	1	3
Margin	4	1	5

Table 4. Table 4: Initial contingency table 𝒏 𝒛 ​ 𝒘 ​ 𝒛 ′ ​ 𝒘 ′ superscript 𝒏 𝒛 𝒘 superscript 𝒛 bold-′ superscript 𝒘 bold-′ \boldsymbol{n^{zwz^{\prime}w^{\prime}}} (see Definition 2.1).

Block	$(1,1)$	$(1,2)$	$(1,3)$	$(2,1)$	$(2,2)$	$(2,3)$	Margin
$(1,1)$	6	0	2	0	0	0	8
$(1,2)$	0	4	0	0	0	0	4
$(2,1)$	6	0	2	3	0	1	12
$(2,2)$	0	4	0	0	2	0	6
Margin	12	8	4	3	2	1	30

Table 5. Table 5: Assessment of the behavior of the criteria with regard to the five desired properties. ✓means that the criterion is validated while ✗ means that the criterion is not fulfilled (or the least efficient compared to the other criteria).

	CARI	CE	NCE	ENMI	coNMI
		(Lomet, 2012)		(Wyse et al., 2017)
1. Proportion of cells misclassified	Moderately	✓	✓	✗	✗
2. Symmetry	✓	✓	✓	✓	✓
3.1 Maximum limit equal to 1	✓	If we use $1-\text{CE}$	✓	✓	✓
\hdashline3.2 Minimum limit equal to 0	Asympto-tically	✗	✓	✓	✓
4. Label switching	✓	✓	✓	✓	✓
5. Execution time	Reasonable	Impossible as soon as $\max (H,L)>9$	Reasonable	✓	✓

Table 6. Table 6: Characteristics of the co-clustering partitions provided by the two procedures on the MovieLens dataset.

	$GS$	Bi-KM1
Number of row partitions	50	56
Number of column partitions	55	49

Equations

$$\frac{a+d}{a+b+c+d}=\frac{a+d}{\left(\genfrac{}{}{0ex}{}{I}{2}\right)},$$

$$\text{ARI}=\frac{\text{Index-Expected Index}}{\text{MaxIndex-Expected Index}}.$$

$$\text{ARI}(\mathbf{z},{\mathbf{z}}^{\prime })$$

$$\displaystyle\text{CARI}((\boldsymbol{z,w}),(\boldsymbol{z^{\prime},w^{\prime}}))=\frac{\sum\limits_{p,q}\binom{n_{p,q}^{zwz^{\prime}w^{\prime}}}{2}-\sum\limits_{p}\binom{n^{zwz^{\prime}w^{\prime}}_{p,\centerdot}}{2}\sum\limits_{q}\binom{n^{zwz^{\prime}w^{\prime}}_{\centerdot,q}}{2}/\binom{I\times J}{2}}{\frac{1}{2}\bigg{[}\sum\limits_{p}\binom{n^{zwz^{\prime}w^{\prime}}_{p,\centerdot}}{2}+\sum\limits_{q}\binom{n^{zwz^{\prime}w^{\prime}}_{\centerdot,q}}{2}\bigg{]}-\bigg{[}\sum\limits_{p}\binom{n^{zwz^{\prime}w^{\prime}}_{p,\centerdot}}{2}\sum\limits_{q}\binom{n^{zwz^{\prime}w^{\prime}}_{\centerdot,q}}{2}\bigg{]}/\binom{I\times J}{2}}.$$

$${\mathbf{n}}^{\mathbf{zw}{\mathbf{z}}^{\prime }{\mathbf{w}}^{\prime }}={\mathbf{n}}^{\mathbf{z}{\mathbf{z}}^{\prime }}\otimes {\mathbf{n}}^{\mathbf{w}{\mathbf{w}}^{\prime }},$$

$$n_{\cdot ,q}^{zw{z}^{\prime }{w}^{\prime }}=n_{\cdot h_q^{\prime }}^{z{z}^{\prime }}\otimes n_{\cdot ,{\ell }_q^{\prime }}^{w{w}^{\prime }}\text{ and }{n}_{p,\cdot }^{zw{z}^{\prime }{w}^{\prime }}=n_{h_p,\cdot }^{z{z}^{\prime }}\otimes n_{{\ell }_p,\cdot }^{w{w}^{\prime }}$$

$$\text{CARI}((\mathbf{z},\mathbf{w}),({\mathbf{z}}^{\prime },{\mathbf{w}}^{\prime }))=\text{CARI}(({\mathbf{z}}^{\prime },{\mathbf{w}}^{\prime }),(\mathbf{z},\mathbf{w})).$$

$${\text{dist}}_{(I,H)\times (J,L)}((\mathbf{z},\mathbf{w}),({\mathbf{z}}^{\prime },{\mathbf{w}}^{\prime }))$$

$$\text{CE}((\mathbf{z},\mathbf{w}),({\mathbf{z}}^{\prime },{\mathbf{w}}^{\prime }))$$

$${\text{dist}}_{I,H}(\mathbf{z},{\mathbf{z}}^{\prime })=1-\underset{\sigma \in \mathfrak{S}(\{1,...,H\})}{\max }\frac{1}{I}\sum _{i,h}{z}_{ih}{z}_{i\sigma (h)}^{\prime }.$$

$${\text{dist}}_{(I,H)\times (J,L)}((\mathbf{z},\mathbf{w}),({\mathbf{z}}^{\prime },{\mathbf{w}}^{\prime }))$$

$$\sum _{i,h}{z}_{ih}{z}_{i\sigma (h)}^{\prime }=\sum _h{\mathbf{n}}_{h\sigma (h)}^{\mathbf{z}{\mathbf{z}}^{\prime }}.$$

$$\text{NCE}((\mathbf{z},\mathbf{w}),({\mathbf{z}}^{\prime },{\mathbf{w}}^{\prime }))=1-\frac{{\text{dist}}_{(I,H)\times (J,L)}((\mathbf{z},\mathbf{w}),({\mathbf{z}}^{\prime },{\mathbf{w}}^{\prime }))}{\frac{1}{H}+\frac{1}{L}-\frac{1}{HL}}$$

$$\text{MI}(\boldsymbol{z},\boldsymbol{z^{\prime}})=\sum\limits_{h,h^{\prime}}P_{h,h^{\prime}}\log\big{(}\frac{P_{h,h^{\prime}}}{P_{h}P_{h^{\prime}}}\big{)},$$

$$\text{where},P_{h,h^{\prime }}=\frac{1}{I}\sum _i^I\text{mathds}{1}_{\{z_i=h,z_i^{\prime }=h^{\prime }\}},P_h=\frac{1}{I}\sum _i^I\text{mathds}{1}_{\{z_i=h\}}\text{ and }{P}_{h^{\prime }}=\frac{1}{I}\sum _i^I\text{mathds}{1}_{\{z_i^{\prime }=h^{\prime }\}}.$$

$$\text{NMI}(\mathbf{z},{\mathbf{z}}^{\prime })=\frac{\text{MI}(\mathbf{z},{\mathbf{z}}^{\prime })}{\max (\mathcal{H}(\mathbf{z}),\mathcal{H}({\mathbf{z}}^{\prime }))},$$

$$\text{where,}\mathcal{H}(\mathbf{z})=-\sum _h{P}_h\log P_h,\text{ and }\mathcal{H}(z^{\prime })=-\sum _{h^{\prime }}{P}_{h^{\prime }}\log P_{h^{\prime }}.$$

$$\text{ENMI}((\mathbf{z},\mathbf{w}),({\mathbf{z}}^{\prime },{\mathbf{w}}^{\prime }))=\text{NMI}(\mathbf{z},{\mathbf{z}}^{\prime })+\text{NMI}(\mathbf{w},{\mathbf{w}}^{\prime }).$$

$$\text{coMI}((\mathbf{z},\mathbf{w});({\mathbf{z}}^{\prime },{\mathbf{w}}^{\prime }))=\sum _{p,q}\frac{n_{p,q}^{zw{z}^{\prime }{w}^{\prime }}}{IJ}\log \left(\frac{n_{p,q}^{zw{z}^{\prime }{w}^{\prime }}IJ}{n_{p,\cdot }^{zw{z}^{\prime }{w}^{\prime }}{n}_{\cdot ,q}^{zw{z}^{\prime }{w}^{\prime }}}\right)$$

$$\text{coNMI}((\mathbf{z},\mathbf{w});({\mathbf{z}}^{\prime },{\mathbf{w}}^{\prime }))=\frac{\text{coMI}((\mathbf{z},\mathbf{w});({\mathbf{z}}^{\prime },{\mathbf{w}}^{\prime }))}{\max (\mathcal{H}(\mathbf{z},\mathbf{w}),\mathcal{H}({\mathbf{z}}^{\prime },{\mathbf{w}}^{\prime }))}$$

$$\mathcal{H}(\mathbf{z},\mathbf{w})=-\sum _p\frac{n_{p,\cdot }^{zw{z}^{\prime }{w}^{\prime }}}{IJ}\log \left(\frac{n_{p,\cdot }^{zw{z}^{\prime }{w}^{\prime }}}{IJ}\right)$$

$$\mathcal{H}({\mathbf{z}}^{\prime },{\mathbf{w}}^{\prime })=-\sum _q\frac{n_{\cdot ,q}^{zw{z}^{\prime }{w}^{\prime }}}{IJ}\log \left(\frac{n_{\cdot ,q}^{zw{z}^{\prime }{w}^{\prime }}}{IJ}\right).$$

$$\text{coMI}((\mathbf{z},\mathbf{w});({\mathbf{z}}^{\prime },{\mathbf{w}}^{\prime }))=\text{MI}(\mathbf{z},{\mathbf{z}}^{\prime })+\text{MI}(\mathbf{w},{\mathbf{w}}^{\prime }).$$

$$m_{CARI}$$

$$\frac{HL+H^{\prime }{L}^{\prime }-HL{H}^{\prime }{L}^{\prime }}{IJ\left(\frac{HL+H^{\prime }{L}^{\prime }}{2}-1\right)+HL+H^{\prime }{L}^{\prime }-HL{H}^{\prime }{L}^{\prime }}+o(1)\le CARI((\text{z},{\text{z}}^{\prime }),(\text{w},{\text{w}}^{\prime }))$$

$$0\le \text{MI}(\text{z},{\text{z}}^{\prime })\le \min \left(\mathcal{H}(\text{z}),\mathcal{H}({\text{z}}^{\prime })\right).$$

$$0\le \text{EMMI}((\text{z},\text{w}),({\text{z}}^{\prime },{\text{w}}^{\prime }))\le 2.$$

$$0\le {\text{dist}}_{I,H}(\mathbf{z},{\mathbf{z}}^{\prime })\le \frac{H-1}{H}.$$

$$1-\frac{1}{H}-\frac{1}{L}+\frac{1}{HL}\le$$

$$\text{and }0\le$$

$$P_{h,h^{\prime }}=\left(\begin{array}{cc}\frac{1-x}{2}& \frac{x}{2}\\ \frac{x}{2}& \frac{1-x}{2}\end{array}\right)\text{ and }{P}_{\ell ,{\ell }^{\prime }}=\left(\begin{array}{cc}\frac{1-y}{2}& \frac{y}{2}\\ \frac{y}{2}& \frac{1-y}{2}\end{array}\right)$$

Full text

The new criterion we developed is based on the Adjusted Rand Index and is called the Co-clustering Adjusted Rand Index named CARI. We also suggest new improvements to existing criteria such as the Classification Error which counts the proportion of misclassified cells and the Extended Normalized Mutual Information criterion which is a generalization of the criterion based on mutual information in the case of classic classifications.

We study these criteria with regard to some desired properties deriving from the co-clustering context. Experiments on simulated and real observed data are proposed to compare the behavior of these criteria.

Keywords: Co-clustering, Adjusted Rand Index, Mutual information, Agreement, Partition

1 Introduction

With the advent of sparse and high-dimensional datasets in statistics, co-clustering has become a topic of interest in recent years in many fields of applications. For example, in the context of text mining, Dhillon et al. (2003) aim at finding similar documents and their interplay with word clusters. In genomics, the objective of Jagalur et al. (2007) is to identify groups of genes whose expression is linked to anatomical sub-structures in the mouse brain. As for Shan & Banerjee (2008) or Wyse et al. (2017), they used co-clustering to obtain groups of users sharing the same movie tastes and to improve recommendation systems. Finally, we can cite a new approach in pharmacovigilance proposed by Keribin et al. (2017) using co-clustering to simultaneously provide clusters of individuals sharing the same drug profile along with the link between drugs and non-detected adverse events.

Originally developed by Hartigan (1975), a co-cluster analysis aims at reducing the data matrix to a simpler one while preserving the information contained in the initial matrix (Govaert & Nadif, 2013). Co-clustering methods may provide a simultaneous partition of two sets $A$ (rows, observations, individuals) and $B$ (columns, variables, attributes). The major advantages of this method consist in drastically decreasing the dimension of clustering and the ease to deal with sparse data (Dhillon et al., 2003).Thanks to co-clustering methods, information about the dependence between rows and columns can be extracted. Furthermore, co-clustering results are easy to interpret through data visualization in terms of blocks.

To assess the performances of a co-clustering method, partitions obtained with the procedure need to be evaluated. Objective criteria are therefore required to measure how close these partitions are to a reference structure.

On the one hand, Charrad et al. (2010) suggest a first solution and linearly extend from clustering several internal validity indices (Dunn index, Baker and Hubert index, Davies and Bouldin index, Calinsky and Harabsz index, Silhouette criterion, Hubert and Levin index, Krzanowski and Lai index and the differential method). Wyse et al. (2017) also extend in the same way, through the use of an external validity index, that is based on the normalized mutual information measure discussed in Vinh et al. (2010).

However, simply defining a linear combination between the index for row partitions and the index for column partitions fails to preserve the co-clustering structure. From a co-clustering point of view, the statistical unity is a cell within a matrix which is characterized simultaneously by its row number and its column number. If the rows are considered separately from the columns through in an index, a wrongly classified cell may be counted twice. On the other hand, Lomet (2012) propose a distance measure particularly designed for assessing agreement between co-clustering partitions. The computation of this index is dependent on the number of partition permutations and this property make its calculation time-consuming. More precisely, the numbers of clusters can barely exceed nine in each direction. Moreover, no mathematical convention is given when the number of clusters of the compared partitions is different.

The aim of the present paper is to adapt the very popular and commonly used Adjusted Rand Index (ARI) developed by Hubert & Arabie (1985) to a co-clustering point of view. In order to challenge other indices and tackle the problem of pairs of partitions with possibly different and high numbers of clusters, this new index takes into account the co-clustering structure while keeping its computation time efficient.

The paper is organized as follows. In the next section, the statistical framework is introduced with the desired properties to evaluate two co-clustering partitions. The new index we developed, which is named the Co-clustering Adjusted Rand Index (CARI) is presented. Furthermore, two indices that have also been proposed for comparing co-clustering partitions are considered and examined from a co-clustering point of view; improvements are suggested each time. In Section 3, theoretical properties of the five criteria are studied. Section 4 is devoted to evaluating and comparing the criteria regarding the desired properties. In Section 5, a real observed data application is presented in which the application of CARI is illustrated. Finally a discussion section ends this paper.

2 Statistical framework

In this section, we present the co-clustering framework and the new external validity index we developed to assess co-clustering results. Other competitor criteria are also detailed and compared with the proposed criterion.

2.1 Co-clustering context

Co-clustering is a method of cross-classification of arrays whose objective is to form a grid of blocks that are as homogeneous as possible inside and the most heterogeneous when they are taken in pairs (see Figure 1). This grid is the Cartesian product of a partition of the rows and a partition of the columns of the matrix.

Co-clustering, which aims at designing a grid of blocks, ought to be contrasted with other types of cross-classifications. For example, a nested classification aims at designing blocks within already formed blocks whereas a bi-clustering aims at designing only the most homogeneous blocks without taking into account possible overlap (see Figure 2 for schematic examples and see Hartigan (1975)).

As explained in Brault & Mariadassou (2015), a difference can be made between co-clustering and the need to separately cluster the rows and columns of a matrix. In the first case, the statistical unit is the cell of the matrix and the objective is to assign it to the relevant block. In the second case, the statistical units are, on the one hand, the lines and on the other hand, the columns: the goal is then to know if the rows (and independently of the columns) are then aggregated in the suitable clusters. In Figure 3, the differences between two co-partitions are schematized: the yellow (resp. red) cells are correctly classified towards rows (resp columns) but misclassified towards columns (resp rows). The orange cells are located at the intersection of rows and columns that do not belong to the same clusters in both co-partitions. In the co-clustering point of view, each yellow, red, and orange cell is misclassified as it does not belong to the same blocks (for example, some of the yellow cells belong to block $(1,1)$ for the first classification and block $(2,1)$ for the second). In the second point of view, only the orange cells which are located at the intersection are totally misclassified. The yellow and red cells are only considered partially misclassified since only the row or the column of which they are at the intersection is misclassified.

Finally, as in any mixture model, the problem of label switching must be taken into account: if two partitions are exactly equal but the cluster numbers are mixed, we must consider that the partitions are identical.

In conclusion, the objective of this work is therefore to propose a suitable criterion that is linked to the rate of misclassified cells of a matrix. It should penalize with the same weight every cell that does not belong to the same block of two co-partitions, and take into account the label switching problem. The desired properties are therefore the following:

The index should be proportional to the number of miss-classified cells. 2. 2.

The index should be symmetrical regarding two co-partitions. 3. 3.

The index should vary between 0 and 1, where 1 means a perfect match and 0 leads to all worst scenarios, including independence. 4. 4.

The index should be independent from label switching: if two co-partitions are equal when the labels are swaped, the score should remain unchanged. 5. 5.

The index should be computable within a reasonable time.

2.2 Rand index (RI) and Adjusted Rand Index (ARI)

The index we developed further is based on commonly used distances in clustering: the Rand Index and the Adjusted Rand Index. Before introducing this new index, we shall summarize the principles and definitions of the latter criteria.

Let two partitions be $\boldsymbol{z}=(z_{1},\ldots z_{i},\ldots,z_{I})$ and $\boldsymbol{z^{\prime}}=(z^{\prime}_{1},\ldots,z^{\prime}_{i},\ldots,z^{\prime}_{I})$ on a set $A$, with Card($A$)=$I$, $\max(\boldsymbol{z})=H$ and $\max(\boldsymbol{z^{\prime}})=H^{\prime}$. In this article, the set $A$ will be further the set of the indices of the rows of a matrix. For example, $\boldsymbol{z}$ denotes an external reference and $\boldsymbol{z^{\prime}}$ denotes a partition derived from a clustering result.

The Rand Index (RI) developed by Rand (1971), is a measure of the similarity between two partitions $\boldsymbol{z}$ and $\boldsymbol{z^{\prime}}$, and is defined as follows:

[TABLE]

where,

• •

$a$ denotes the number of pairs of elements that are placed in the same cluster in $\boldsymbol{z}$ and in the same cluster in $\boldsymbol{z^{\prime}}$,

• •

$b$ denotes the number of pairs of elements in the same cluster in $\boldsymbol{z}$ but not in the same cluster in $\boldsymbol{z^{\prime}}$,

• •

$c$ denotes the number of pairs of elements in the same cluster in $\boldsymbol{z^{\prime}}$ but not in the same cluster in $\boldsymbol{z}$,

• •

$d$ denotes the number of pairs of elements in different clusters in both partitions.

The values $a$ and $d$ can be interpreted as agreements, and $b$ and $c$ as disagreements. To compute all these values, a contingency table can be introduced. Let $\boldsymbol{n^{zz^{\prime}}}=(n_{h,h^{\prime}}^{zz^{\prime}})_{H\times H^{\prime}}$ be the matrix where $n_{h,h^{\prime}}^{zz^{\prime}}$ denotes the number of elements of set $A$ which belong to both cluster $h$ and cluster $h^{\prime}$. The row and column margins $n_{h,\centerdot}^{zz^{\prime}}$ and $n_{\centerdot,h^{\prime}}^{zz^{\prime}}$ denote respectively the number of elements in cluster $h$ and $h^{\prime}$. We thus find the following correspondence, (Albatineh et al., 2006):

• •

a=\sum\limits_{h}^{H}\sum\limits_{h^{\prime}}^{H^{\prime}}\binom{n^{zz^{\prime}}_{h,h^{\prime}}}{2}=\frac{\sum\limits_{h}^{H}\sum\limits_{h^{\prime}}^{H^{\prime}}\big{(}n^{zz^{\prime}}_{h,h^{\prime}}\big{)}^{2}-I}{2},

• •

b=\sum\limits_{h}^{H}\binom{n^{zz^{\prime}}_{h,\centerdot}}{2}-a=\frac{\sum\limits_{h}^{H}(n^{zz^{\prime}}_{h,\centerdot})^{2}-\sum\limits_{h}^{H}\sum\limits_{h^{\prime}}^{H^{\prime}}\big{(}n^{zz^{\prime}}_{h,h^{\prime}}\big{)}^{2}}{2},

• •

c=\sum\limits_{h^{\prime}}^{H^{\prime}}\binom{n^{zz^{\prime}}_{\centerdot,h^{\prime}}}{2}-a=\frac{\sum\limits_{h^{\prime}}^{H^{\prime}}(n^{zz^{\prime}}_{\centerdot,h^{\prime}})^{2}-\sum\limits_{h}^{H}\sum\limits_{h^{\prime}}^{H^{\prime}}\big{(}n^{zz^{\prime}}_{h,h^{\prime}}\big{)}^{2}}{2},

• •

$d=\binom{I}{2}-(a+b+c)$,

This symmetric index lies between [math] and $1$ and takes the value of $1$ when the two partitions perfectly agree up to a permutation. Thus, by comparing pairs of elements, this index does not need to review all the permutations of studied partitions, making its computation efficient.

The expected value of the Rand Index for two random partitions does not take a constant value and its values are concentrated in a small interval close to $1$ (Fowlkes & Mallows (1983)). The Adjusted Rand Index (ARI) proposed by Hubert & Arabie (1985) enables us to overcome such drawbacks. This corrected version assumes the generalized hypergeometric distribution as the model of randomness, that is to say partitions are chosen randomly so that the number of elements in the clusters can be fixed. The general form of this index which is the normalized difference between the Rand Index and its expected value under the generalized hypergeometric distribution assumption, is as follows:

[TABLE]

The quantity MaxIndex in Equation (1) is the maximum value of the index regardless of the marginal numbers. This index is bounded by $1$, and takes this value when the two partitions are equal up to a permutation. It can also take negative values, which corresponds to less agreement than expected by chance.

From Equation (1), Warrens (2008) shows the ARI can be written in this way:

[TABLE]

Like the RI, the ARI is symmetric, that is to say ARI$(\boldsymbol{z},\boldsymbol{z^{\prime}})$=ARI$(\boldsymbol{z^{\prime}},\boldsymbol{z})$. Indeed, the contingency table associated with ARI$(\boldsymbol{z^{\prime}},\boldsymbol{z})$, is $t(\boldsymbol{n^{zz^{\prime}}})$, where $t$ denotes the transpose of a matrix. Besides, in Equation (2.2) in which the ARI is defined, the margins of the contingency table act in a symmetric way. That is why, while considering $\boldsymbol{n^{zz^{\prime}}}$ or its transpose matrix $t(\boldsymbol{n^{zz^{\prime}}})$, the ARI remains unchanged. This remark will be particularly interesting in the next section, when the new index we developed is studied.

2.3 Co-clustering Adjusted Rand Index (CARI)

We extend the Adjusted Rand Index to a co-clustering point of view in order to compare two co-clustering partitions which define blocks, and no longer clusters.

Let $\boldsymbol{z}=(z_{1},\ldots,z_{i},\ldots z_{I})$ and $\boldsymbol{z^{\prime}}=(z^{\prime}_{1},\ldots,z^{\prime}_{i},\ldots,z^{\prime}_{I})$ be two partitions on a set $A$. Let $\boldsymbol{w}=(w_{1},\ldots,w_{j},\ldots,w_{J})$ and $\boldsymbol{w^{\prime}}=(w^{\prime}_{1},\ldots,w^{\prime}_{j},\ldots,w^{\prime}_{J})$ be two partitions on a set $B$. $(\boldsymbol{z},\boldsymbol{w})$ and $(\boldsymbol{z^{\prime}},\boldsymbol{w^{\prime}})$ are two co-clustering partitions on the set $A\times B$ where an observation is denoted by $x_{ij},i=1,\ldots,I;j=1,\ldots,J$, with Card($A\times B)=I\times J$. Notice that $\boldsymbol{z}$ and $\boldsymbol{z^{\prime}}$ are called row partitions. Similarly, $\boldsymbol{w}$ and $\boldsymbol{w^{\prime}}$ are called column partitions. We also define the associated binary matrices $z=(z_{ih})_{I\times H}$ and $w=(w_{j\ell})_{J\times L}$ which $z_{ih}$ indicates if the row $i$ belongs to cluster $h$ and $w_{jl}$ indicates if the column $j$ belongs to cluster $\ell$. We use the same notations to denote the vector or the matrix, as it is commonly used in mixture models.

Definition 2.1**.**

The contingency table $\boldsymbol{n^{zwz^{\prime}w^{\prime}}}=(n_{p,q}^{zwz^{\prime}w^{\prime}})$ where $1\leq p\leq HL$ and $1\leq q\leq H^{\prime}L^{\prime}$, is defined so that $n_{p,q}^{zwz^{\prime}w^{\prime}}$ denotes the number of observations of set $A\times B$ belonging to block $p$ (related to a pair $(h,\ell)$ defined by $(\boldsymbol{z},\boldsymbol{w})$) and block $q$ (related to pair $(h^{\prime},\ell^{\prime})$ defined by $(\boldsymbol{z^{\prime}},\boldsymbol{w^{\prime}})$).

The contingency table can be seen as a block matrix which consists of $H\times H^{\prime}$ blocks of size $L\times L^{\prime}$ (see Table 1).

Notice that a bijection can be defined between the index $p$ of the rows of the contingency table, and the block $(h,\ell)$ defined by $(\boldsymbol{z},\boldsymbol{w})$ (see Appendix A.1).

An analogous correspondence is defined for the index $q$ and the block $(h^{\prime},\ell^{\prime})$ defined by $(\boldsymbol{z^{\prime}},\boldsymbol{w^{\prime}})$. Thus the notation ($h_{p}$ $\ell_{p}$) and ($h^{\prime}_{q}$ $\ell^{\prime}_{q}$) could be used. We will see afterwards, this trick enables us to describe $\boldsymbol{n^{zwz^{\prime}w^{\prime}}}$ in a convenient way.

Definition 2.2**.**

Given two co-clustering partitions $(\boldsymbol{z},\boldsymbol{w})$ and $(\boldsymbol{z^{\prime}},\boldsymbol{w^{\prime}})$ and their corresponding contingency table $\boldsymbol{n^{zwz^{\prime}w^{\prime}}}$ specified in Definition 2.1, the Co-clustering Adjusted Rand Index (CARI) is defined as follows:

[TABLE]

Like the ARI, this index is symmetric and takes the value of $1$ when the pairs of co-partitions perfectly agree up to a permutation. But unlike the index proposed in Lomet (2012) with which, we will compare our work in Section 3 and 4, no mathematical convention is needed when the number of clusters is different in partitions. Moreover, it does not based on the permutations of partitions and can therefore be easily computed even if the number of row clusters or column clusters exceeds nine. That said, the complexity to compute $\boldsymbol{n^{zwz^{\prime}w^{\prime}}}$ is still substantial.

Fortunately, we manage to exhibit a link between $\boldsymbol{n^{zwz^{\prime}w^{\prime}}}$, $\boldsymbol{n^{zz^{\prime}}}$ and $\boldsymbol{n^{ww^{\prime}}}$ which makes the computation of the CARI much faster and competitive when the numbers of clusters are high. This link is presented in the next theorem.

Theorem 2.3**.**

We have the following relation,

[TABLE]

where $\otimes$ denotes the Kronecker product between two matrices, $(\boldsymbol{z},\boldsymbol{w})$ and $(\boldsymbol{z^{\prime}},\boldsymbol{w^{\prime}})$ are two co-clustering partitions, $\boldsymbol{n^{zwz^{\prime}w^{\prime}}}$ is the corresponding contingency table specified in Definition 2.1, $\boldsymbol{n^{zz^{\prime}}}$ and $\boldsymbol{n^{ww^{\prime}}}$ are defined in Section 2.2.

The proof of this theorem is presented in Appendix A.1.

Thanks to this property, the contingency table $\boldsymbol{n^{zwz^{\prime}w^{\prime}}}$ can be computed more efficiently as we will see in the next sections. Moreover, even if the Kronecker product is not commutative, it behaves well with both the transpose operator and the margins of a matrix (sum of the observations according to the rows or the columns) and the initial properties of CARI are kept:

Corollary 2.4**.**

For all $(p,q)\in\{1,\ldots,HL\}\times\{1,\ldots,H^{\prime}L^{\prime}\}$, we have the relations between the margins,

[TABLE]

where $(h_{p},\ell_{p})$ (resp. $(h^{\prime}_{q},\ell^{\prime}_{q})$) are the coordinates in the co-clustering partition $(\textbf{z},\textbf{w})$ (resp. $(\textbf{z}^{\prime},\textbf{w}^{\prime})$) associated with the block $p$ (resp. $q$).** 2. 2.

The CARI criterion associated with the contingency table $\boldsymbol{n^{zwz^{\prime}w^{\prime}}}$ defined as in Equation (3) remains symmetric, that is to say,

[TABLE]

The proof of this corollary is presented in Appendix A.2.

In the further sections, the contingency table $\boldsymbol{n^{zwz^{\prime}w^{\prime}}}$ is now defined by Equation (3).

To illustrate the properties of CARI, we developed several simple examples. Consider the following pairs of co-partitions $(\boldsymbol{z,w})=((1,1,3,2),(1,2,1,4,3))$ and $(\boldsymbol{z^{\prime},w^{\prime}})=((2,2,1,3),(2,1,2,3,4))$ which are equal up to a permutation. The contingency table (see Table 2) associated with CARI$((\boldsymbol{z,w}),(\boldsymbol{z^{\prime},w^{\prime}}))$ has a size of $(3\times 4,3\times 4)$. Thus, the CARI$((\boldsymbol{z,w}),(\boldsymbol{z^{\prime},w^{\prime}}))$ criterion behaves in the desired way and is equal $\frac{11-121/190}{1/2\times 22-121/190}=1$.

Let us now consider the following co-partitions $(\boldsymbol{z,w})=((1,2,2,2,1),(1,1,2,1,1,2))$ and $(\boldsymbol{z^{\prime},w^{\prime}})=((1,1,2,1,1),(1,1,2,1,3,2))$. Note that partitions $\boldsymbol{w}$ and $\boldsymbol{w^{\prime}}$ do not have the same number of clusters. The initial contingency tables related to ARI$(\boldsymbol{z,z^{\prime}})$, ARI$(\boldsymbol{w,w^{\prime}})$ and CARI$((\boldsymbol{z,w}),(\boldsymbol{z^{\prime},w^{\prime}}))$ are described in Table 3 and in Table 4. Each cell of Table 4 represents the number of observations which belongs to both blocks in the first and the second co-clustering partitions, for all blocks. As announced, we observe that $\boldsymbol{n^{zwz^{\prime}w^{\prime}}}=\boldsymbol{n^{zz^{\prime}}}\otimes\boldsymbol{n^{ww^{\prime}}}.$ and the value of CARI is equal approximatively 0.2501.

2.4 Classification error

The classification distance presented in Lomet (2012) aims at studying the misclassification rate of the observations in the blocks. Given two co-clustering partitions, $(\boldsymbol{z},\boldsymbol{w})$ and $(\boldsymbol{z^{\prime}},\boldsymbol{w^{\prime}})$, and with the notations used in Section 2.3, the classification error is defined as follows,

[TABLE]

where $\mathfrak{S}(\{1,...,H\})$ denotes the set of permutations on the set $\{1,\ldots,H\}$.

Remark**.**

The previous definition assumes that the two classifications have the same number of clusters; if not, just assume that there are empty clusters and take $H=H^{\prime}$.

The classification error (CE) is then defined when the cost function measures the difference between two co-clustering partitions $(\boldsymbol{z},\boldsymbol{w})$ and $(\boldsymbol{z^{\prime}},\boldsymbol{w^{\prime}})$:

[TABLE]

The classification error produces a value between [math] and $1$. Thus, the observation $x_{ij}$ is not in block $(h,\ell)$ if row $i$ is not in cluster $h$ or if column $j$ is not in cluster $\ell$. When a column is improperly classified, all the cells of this column are penalized, and the classification error is increased by $\frac{1}{J}$.

Furthermore, the distance related to the row partitions can also be defined as follows:

[TABLE]

When the partitions do not include the same number of clusters, a suitable convention we can propose, is to consider $H$ as the maximal number of clusters and the created additional clusters are assumed to be empty. The computation of this distance when $H$ is higher than nine, remains difficult as the order of the set $\mathfrak{S}(\{1,...,H\})$ is $H!$.

In a symmetric way, the distance related to the column partitions is denoted by $\text{dist}_{\scriptsize{J,L}}$.

Lomet (2012) shows that the classification error could be expressed in terms of the distance related to the row partitions and the distance related to the column partitions:

[TABLE]

As we prove in Section 3, computing a quantity involving the set of permutations is impossible as soon as $H$ or $L$ are greater than 10. However, this problem can be seen as an assignment problem which allows the use of optimization algorithms such as the Hungarian algorithm proposed by Kuhn (1955). Indeed, we observe that we can rewrite Equation (4) in the following way:

[TABLE]

Optimizing the criterion in Equation (4) means to search for a permutation of the columns of the matrix $\boldsymbol{n}^{\boldsymbol{zz^{\prime}}}$ so that the diagonal can have the largest possible sum. The rows in the contingency matrix $\boldsymbol{n}^{\boldsymbol{zz^{\prime}}}$ can be interpreted as agents, and the columns as tasks to be performed; each cell represents each agent’s capacity to perform a task. The aim of the assignment problem is to assign one task to an agent so that the overall capacities can be optimized.

Thanks to Reformulation (6) of the problem, we can improve the conjecture proposed by Brault (2014) demonstrated in Section 3.2: the CE criterion has a strictly positive lower bound depending on $H$ and $L$. We propose to redefine the criterion in order to satisfy the desired characteristics:

[TABLE]

where $\text{dist}_{\scriptsize{(I,H)\times(J,L)}}$ is defined in Equation (5).

Remark**.**

Another solution could have been to normalize the distance related to the row partition and the distance related to the column partition independently, then to use Formula (5). However, this choice would create an asymmetry between the roles of the row and column classifications. We rather aim at designing a criterion which counts the number of misclassified cells and is between the value of 0 and the value of 1.

2.5 Extended Normalized Mutual Information

Information theory is another approach to compare the difference in information between two partitions. The Normalized Mutual Information (see for example Linfoot (1957), Knobbe & Adriaans (1996), Quinlan (1986)) is a popular measure in this field. Originally, the Mutual Information (MI) between two partitions $\boldsymbol{z}=(z_{1},\ldots,z_{I})$ and $\boldsymbol{z^{\prime}}=(z^{\prime}_{1},\ldots,z^{\prime}_{I})$ on a same set $A=\{O_{1},\ldots,O_{I}\}$ (with $\max(z)=H$ and $\max(z^{\prime})=H^{\prime}$) is defined as follows:

[TABLE]

[TABLE]

As the measure has no specific upper bound, the mutual information is normalized. This task may be performed in many ways (see Pfitzner et al. (2009)), but Wyse et al. (2017) choose the following normalization to define their co-clustering index:

[TABLE]

[TABLE]

The proposed measure by Wyse et al. (2017) to compare two co-clustering partitions $(\boldsymbol{z}=(z_{1},\ldots,z_{I})$, $\boldsymbol{w}=(w_{1},\ldots,w_{J})$) and ($\boldsymbol{z^{\prime}}=(z^{\prime}_{1},\ldots,z^{\prime}_{I})$, $\boldsymbol{w^{\prime}}=(w^{\prime}_{1},\ldots,w^{\prime}_{J})$) on a set $A\times B$ (with Card$(A)=I$ and Card$(B)=J$) is based on a linear combination of the normalized mutual information of $\boldsymbol{z}$ and $\boldsymbol{z^{\prime}}$, and the normalized mutual information of $\boldsymbol{w}$ and $\boldsymbol{w^{\prime}}$:

[TABLE]

The maximum value of this index is equal 2 when the partitions perfectly match up to a permutation and is equal 0 when the correspondence between them is extremely weak (see Corollary 3.7). However, extending the index in this way fails to preserve the co-clustering structure (see Section 4.1). We therefore propose an adaptation according to the philosophy of the latent block model. We propose the following formula for the new criterion we developed, given two co-partitions, $(\boldsymbol{z},\boldsymbol{w})$ and $(\boldsymbol{z}^{\prime},\boldsymbol{w}^{\prime})$:

[TABLE]

and its standardized version is as follows:

[TABLE]

[TABLE]

Thanks to Theorem 2.3 and Corollary 2.4, the criterion can be computable in polynomial time.

Proposition 2.5**.**

Given two co-clustering partitions $(\boldsymbol{z},\boldsymbol{w})$ and $(\boldsymbol{z^{\prime}},\boldsymbol{w^{\prime}})$, we have:

[TABLE]

The proof is based on Theorem 2.3 and Corollary 2.4.

Remark**.**

The difference between criteria $ENMI/2$ and $coNMI$ is only based on the normalization; in particular, as $\mathcal{H}(\boldsymbol{z},\boldsymbol{w})=\mathcal{H}(\boldsymbol{z})+\mathcal{H}(\boldsymbol{w})$, the criteria are equal as soon as $\mathcal{H}(\boldsymbol{z})=\mathcal{H}(\boldsymbol{z^{\prime}})=\mathcal{H}(\boldsymbol{w})=\mathcal{H}(\boldsymbol{w^{\prime}})$.

3 Theoretical properties of the indices

In this section, we present theoretical properties of the indices such as the complexity and the boundaries of each criterion.

3.1 Complexities

In this section, we present the complexity for the computation of each criterion.

Proposition 3.1**.**

The complexities for the computation of each criterion are the following:

• •

The CARI criterion has the complexity $\mathcal{O}\left(\max\left(HH^{\prime}LL^{\prime},I,J\right)\right)$.**

• •

The ENMI and coNMI criteria have the complexity $\mathcal{O}\left(\max\left(HH^{\prime},I,LL^{\prime},J\right)\right)$.**

• •

The CE criterion has the complexity $\mathcal{O}\left(\max\left(H!HI,L!LJ\right)\right)$ where $H!$ is the factorial of $H$.

• •

The NCE criterion has the complexity $\mathcal{O}\left(\max(H^{4},H^{\prime 4},I,L^{4},L^{\prime 4},J)\right)$.**

The proofs are available in Appendix A.4. In particular, we can observe that $CE$ is usually incalculable as soon as $H$ or $L$ is greater than 10 ($10!\approx 3\,628\,800$). Note that most of the steps in these procedures can be easily parallelized.

Remark**.**

For the complexity of the NCE criterion, the used complexity for the Hungarian algorithm is that proposed by Kuhn (1955). In some cases, this complexity can be improved111In the simulations, we use the R-package lpSolve proposed by Berkelaaret al. (2020) available on CRAN..

3.2 Boundaries

In this part, we study the boundaries of each criterion. For the CARI criterion, we only propose an upper bound of the lower bound and we conjecture that it is the upper bound too.

Proposition 3.2**.**

If the upper bound is denoted by $m_{CARI}$ for every partition z, $\textbf{z}^{\prime}$, w and $\textbf{w}^{\prime}$, we have:

[TABLE]

The proof is available in Appendix A.5. From this result, we conjecture that:

Conjecture 3.3**.**

For every partition z, $\textbf{z}^{\prime}$, w and $\textbf{w}^{\prime}$, we have

[TABLE]

In particular, if $H$, $H^{\prime}$, $L$ and $L^{\prime}$ are fixed, the lower bound is negative and tends to [math] when $IJ$ tends to $+\infty$.

The following boundaries of the ENMI criterion are as follows:

Proposition 3.4**.**

For every partition z and $\textbf{z}^{\prime}$, we have

[TABLE]

A proof is avaiblable in Appendix A.5. This result can be obtained as soon as we consider that the $MI$ is a Kullback-Leibler divergence between the empirical distribution of the joint distribution and the product of the empirical marginal distributions. With this result, we can obtain the boundaries proposed by Wyse et al. (2017) in their article:

Corollary 3.5**.**

For every partition z, $\textbf{z}^{\prime}$, w and $\textbf{w}^{\prime}$, we have

[TABLE]

Finally, we also have boundaries of the CE criterion:

Proposition 3.6**.**

In the case where $H=\max(H,H^{\prime})$ and for each partitions z and $\textbf{z}^{\prime}$, we have:

[TABLE]

Moreover, the upper limit is reached as soon as there are $H$ permutations $\left(\sigma_{1},\ldots,\sigma_{H}\right)$ so that $\left(\boldsymbol{n}_{h,\sigma_{h^{\prime}}\left(h\right)}^{\boldsymbol{zz^{\prime}}}\right)_{1\leq h,h^{\prime}\leq H}$ forms a partition of the cells of the matrix $\boldsymbol{n}^{\boldsymbol{zz^{\prime}}}$ and the values $\sum_{h=1}^{H}\boldsymbol{n}_{h,\sigma_{h^{\prime}}\left(h\right)}^{\boldsymbol{zz^{\prime}}}$ are equal.

We can refine the result when $I$ is not divisible by $H^{2}$ (see for example Appendix B.1 from Brault (2014)). We propose a complete proof based on Equation (6) in Appendix A.5.

Corollary 3.7**.**

For every partition z, $\textbf{z}^{\prime}$, w and $\textbf{w}^{\prime}$ such that $H\geq H^{\prime}$ and $L\geq L^{\prime}$, we have:

[TABLE]

where 1 is a perfect match and the lower bound is reached by all worst scenarios, including independence.

4 Criteria for the desired characteristics

In this section, we consider each point of the desired characteristics set out in Section 2.1.

4.1 Criteria facing the co-clustering context

Co-clustering a matrix and clustering its rows and its columns in a separated way are two different tasks: in co-clustering methods, the statistical unity is a cell of the matrix and no longer a row or a column. A cell is misclassified as soon as it does not belong to the relevant block. A cell does not belong to the suitable block if and only if its row or its column does not belong to the correct cluster. In this section, a toy example where all calculations are possible, is presented. More general simulations are also proposed.

Toy example:

The following simple configurations are considered to illustrate this issue:

[TABLE]

where $x\in[0;1/2]$ and $y\in[0;1/2]$ represent the proportion of misclassified rows and columns respectively and the number of misclassified cells is $x+y-xy$.

The following quantities are equal in the designed model:

[TABLE]

and $\text{coNMI}((\boldsymbol{z},\boldsymbol{w});(\boldsymbol{z}^{\prime},\boldsymbol{w}^{\prime}))=\text{ENMI}((\boldsymbol{z},\boldsymbol{w});(\boldsymbol{z}^{\prime},\boldsymbol{w}^{\prime}))/2$.

The ENMI criterion is based on the sum between the NMI criterion on the rows and on the columns. In Figure 4, two configurations are shown where the ENMI criterion takes the value of 1 in the two cases, when the proportion of well classified cells is respectively 50% (on the left, $(x,y)=(0.5,0)$) and approximately 21% (on the right, $(x,y)\approx(0.1100279,0.1100279)$)222To find the values in Figure 4, we took Configuration (7) and looked for the pairs $(x,y)$ so that the ENMI criterion equals $1\pm 10^{-16}$ by the dichotomy method. As a result, the proportion of misclassified cells and the value of the CARI criterion are slightly approximated.. For the left case, the CARI criterion equals $1/3\approx 0.33$ and for the right case, it equals approximately 0.53.

Thus, for different values of the rate of misclassified cells, the ENMI value remains unchanged, whereas the CARI criterion is doubtless more sensitive.

To go further, Figure 5 shows the values of CARI (in purple), $1-\text{CE}$ (in blue), NCE (in green) and ENMI divided by 2, which is equal to coNMI (in red), according to the proportion of well classified cells for Configuration (7) and for $I=100000$ and $J=100000$. We observe that NCE and CE follow a straight line and are therefore proportional to the number of misclassified cells. Moreover, we observe a lower variance for the CARI criterion than for the ENMI criterion; in particular, on the x-axis (for a fixed criterion value, the possible proportions are smaller for the CARI criterion).

Simulations:

To compare the criteria in more general cases, we propose the following simulation plan for $(I,J)=(100k,50k)$ with $k\in\{1,2\}$, $H$ and $L$ in the set $\{2,\ldots,8\}$:

Simulation of a partition z with $H$ clusters on $I$ rows: for each row $i\in\{1,\ldots,I\}$ uniform simulation of a value in the set $\{1,\ldots,H\}$. Identical simulation for the partition w with $L$ clusters on $J$ columns. 2. 2.

Definition of $\left(\textbf{z}^{(0)},\textbf{w}^{(0)}\right)=\left(\textbf{z},\textbf{w}\right)$. 3. 3.

For $c\in\{1,\ldots,10\,000\}$, do:

1. (a)

Simulation of a random variable following a balanced Bernoulli law $Y\sim\mathcal{B}(1/2)$:

• •

If $Y=0$, let us note $\{u_{1},u_{2},\ldots,u_{\Gamma}\}$ the ordered set of cluster numbers of the matrix $\textbf{z}^{(c-1)}$ and define $\left(\textbf{z}^{(c)},\textbf{w}^{(c)}\right)=\left(\textbf{z}^{(c-1)},\textbf{w}^{(c-1)}\right)$. Simulation of an index of the partition $\textbf{z}^{(c)}$ following an uniformly variable and simulation of a new value for its cluster in the set $\{u_{1},u_{2},\ldots,u_{\ell}\}$ following the weights:

[TABLE]

• •

If $Y=1$, identical procedure except that the drawing is done in the partition of columns $\textbf{w}^{(c)}$. 2. (b)

Computation of the values of each criterion between co-partitions $\left(\textbf{z},\textbf{w}\right)$ and $\left(\textbf{z}^{(c)},\textbf{w}^{(c)}\right)$. 3. (c)

If $\text{NCE}\left((\boldsymbol{z},\boldsymbol{w}),\left(\textbf{z}^{(c)},\textbf{w}^{(c)}\right)\right)<0.01$, break.

Figure 6 shows the values obtained for each criterion as a function of the value obtained by the $NCE$ criterion.

Figure 6 shows that the criterion $CE$ is sensitive to the number of clusters of each partition. The use of the standardized criterion NCE as a reference was therefore preferred. The CARI, ENMI and coNMI criteria do not quite follow the blue line, which means that the values are not perfectly proportional to the number of misclassified cells. Note that the criterion CARI tends to be more penalizing than the other criteria (since its values are below the blue line) but has a lower variance. In contrast, 1-CE, ENMI/2 and coNMI seem to penalize less strongly the proportions of misclassified cells. For the CE criterion, different groups corresponding to different couples $(H,L)$ appear. The comparison of the values obtained can therefore only be made between co-partitions with identical numbers of blocks. Finally, the ENMI/2 and coNMI criteria are more dispersed than the other criteria and can take a large number of values under the blue line. We observe on the graphs that the same value of the coNMI and ENMI/2 criteria corresponds to a large possibility of proportion of poorly classified cells.

4.2 Symmetry

All the criteria are symmetrical.

The proof of this property regarding the CARI criterion is detailed in Corollary 2.4.

For the CE and NCE criteria, the formula is based on the inverse permutation: as we seek the maximum on all possible permutations, we obtain the result.

Finally, the ENMI and coNMI criteria inherit the symmetry property from the MI criterion.

4.3 Boundaries

According to the results presented in Section 3.2, the bound of 1 is reached for two identical co-partitions for the CARI, NCE, coNMI, 1-CE and ENMI/2 criteria.

In contrast, the bound of 0 is reached in the worst cases for the NCE, ENMI and coNMI criteria. In the case of the CARI criterion, the lower bound is slightly negative and tends towards 0 when $I$ and $J$ tend to infinity. On the other hand, the lower bound of the 1-CE criterion is strictly positive as soon as $H$ and $L$ are fixed.

4.4 Criteria facing switching label

In classification, clusters are represented by numbers: individuals with the same number are then in the same cluster. The choice of number is generally arbitrary and two partitions can be identical except for a permutation of numbers: this is what we call label switching. It is therefore important that the criterion does not depend on the choice of cluster numbers.

The CARI, ENMI and coNMI criteria are not influenced by switching label as they add up on all the clusters without an imposed order. Changing the order of the clusters does not change the result. In addition, by definition, the CE and NCE criteria seek the optimal value on all permutations. They finally do not to be sensitive to label switching.

4.5 Time comparison

In this part, we compare the execution times of the different implementations on concrete scenarios.

The complexity of the three indices related to the number of observations and the number of clusters is assessed. For this purpose, the procedure is run with $N=1~{}000$ iterations considering two situations $(I,J)=(250,250)$ observations and $(I,J)=(500,500)$ observations when the number of clusters varies as follows, $(H,L)\in\{(5,5),(7,7),(9,9)\}$. The results obtained with two computer configurations are presented in Figure 7. We observe that the elapsed time computation in log scale of the ENMI is the smallest and seems not to be sensitive to the number of clusters or observations. This optimistic result is notably explained by the fact that the ENMI criterion ignores the co-clustering structure in a pair of partitions. The CARI criterion which takes into account this structure, also behaves as desired whatever the number of clusters or observations. For further information, the elapsed time computation of the CARI criterion on a run of the procedure with $N=2000$, $(I,J)=(2000,2000)$ and $(H,L)=(20,20)$, is three seconds on average, which is reasonable in a high-dimensional co-clustering context. On the contrary, the time computation of the CE significantly increases with the number of clusters, which illustrates its dependence on the factorial of this quantity.

4.6 Comparative study

To conclude this part, we present in Table 5 a summary of the previous results.

We observe that the CE criterion only checks three properties and cannot be computed in a reasonable execution time (which makes its use often obsolete). The improvement we suggest, namely NCE, is the one that satisfies the most of the properties. The ENMI and coNMI criteria verify the last four properties (in particular they are the fastest in computation time) but offer results less in agreement with the philosophy of co-clustering. Finally, the CARI criterion proposed in this article, does not fully verify all the properties but offers an interesting alternative to the NCE criterion (especially when the number of cells is very large).

5 Real data application

In this study, we focus on the MovieLens dataset which includes 100 000 ratings of films from 1 to 5 supplied by 943 users and involves 1682 films 333http://grouplens.org/datasets/movielens/. The studied data matrix (the rows correspond to users and the columns deal with the films) is represented in Figure 8. Notice that if data are missing, they are represented by the value 0. In this section, we aim at comparing co-clustering partitions with high number of clusters, estimated from two methods: the greedy algorithm GS proposed by Wyse et al. (2017) and the Bi-KM1 procedure developed by Robert (2017) and implemented in the R-package bikm1 (Robert, 2020) available on CRAN. The goal is not to study the methods but to compare the proposed solutions.

Thus, the classification goal is to propose groups of individuals sharing the same way to rate specific categories of films. This task is particularly interesting to design recommendation systems.

The co-clustering partitions estimated from the studied real dataset and obtained by the two procedures, are provided by the authors (see Table 6). Both co-clustering partitions have the same order of magnitude regarding the number of row and column clusters. More precisely, the column clusters could correspond to one dominant category of films such as drama or thriller, but it is not the focus of this section. As the Extended Normalized Mutual Information seems not to be relevant in a co-clustering context, and the * classification error* proposed by Lomet (2012) cannot be computed in practice, in this case (as the number of row and column clusters is broadly higher than $8$), we easily computed the Co-clustering Adjusted Rand Index between the co-clustering partitions provided by the two procedures. The CARI’s value is equal 0.3513 and its weak value shows that the provided solutions are quite far from each other. Like any index, its reach is limited in a real data context, as most of times, the reference partitions are often unknown. But in a context where there are some a priori information, thanks to this index, we can easily compare co-clustering methods. Furthermore, this type of index remains essential to assess the performances of a method in a simulation study.

6 Discussion

In this article, criterion CARI based on the Adjusted Rank Index was introduced. We have shown that it can be computed within a reasonable time, thanks to the use of kronecker products. Its behavior was compared to two criteria already used: the Classification Error used by Lomet (2012) and the Extended Normalized Mutual Information introduced by Wyse et al. (2017).

We have shown that the classification error criterion varied according to the number of clusters. The proposed implementation was therefore computationally impossible as soon as the number of clusters exceeds 10. We suggested a new version which allows an estimation independent from the number of clusters. We have proposed a faster implementation by reformulating the issue in terms of an assignment problem.

We have shown that, conceptually, the criterion ENMI does not take into account the cell as a statistical individual. A small modification on normalization enables us to propose the coNMI criterion which adresses this weakness.

We then compared these criteria regarding five properties that we considered important to measure the agreement between two co-classifications. We have observed that only the NCE criterion offers a value proportional to the number of misclassified cells. The CARI criterion seems to slightly penalize more than the other criteria, but the values seem close to the NCE values. Furthermore, experiments underline the fact that the ENMI and coNM criteria could give the same value for configurations with very different proportions of misclassified cells. Finally, we have shown that all criteria can be computed within a reasonable time except criterion CE.

Appendix A Appendix

A.1 Proof of Theorem 2.3

Theorem 2.3. We have the following relation,

[TABLE]

where $\otimes$ denotes the Kronecker product between two matrices, $(\boldsymbol{z},\boldsymbol{w})$ and $(\boldsymbol{z^{\prime}},\boldsymbol{w^{\prime}})$ are two co-clustering partitions, $\boldsymbol{n^{zwz^{\prime}w^{\prime}}}$ is the corresponding contingency table defined according to Definition 2.1, $\boldsymbol{n^{zz^{\prime}}}$ and $\boldsymbol{n^{ww^{\prime}}}$ are defined in Section 2.2.

Proof. Let recall the definition of the Kronecker product. Let $\boldsymbol{A}=(a_{i,j})$ be a matrix of size $H\times H^{\prime}$ and $\boldsymbol{B}$ be a matrix of size $L\times L^{\prime}$. The Kronecker product is the matrix $\boldsymbol{A\otimes B}$ of size $H\times L$ by $H^{\prime}\times L^{\prime}$, defined by successive blocks of size $L\times L^{\prime}$. The block of the index $i,j$ is equal $a_{i,j}\times\boldsymbol{B}$:

[TABLE]

We started by noting a common trick used in computer science. Indeed, for all $p\in\{1,\ldots,HL\}$, the associated pair $(h,\ell)$ denoting a block of $(\boldsymbol{z,w})$, is respectively the quotient plus 1 and the remainder plus 1 of the Euclidean division of $(p-1)$ by $L$. In other words, we have:

[TABLE]

We can easily deduce that there is a bijection between each index $p$ and the pairs $(h,\ell)$. In the same way, the assertion is valid for $q$ and the pairs $(h^{\prime},\ell^{\prime})$.

The next lemma is the last step before proving the final result:

Lemma A.1**.**

For all pairs of indices $p$ and $q$ associated respectively with blocks $(h,\ell)$ and $(h^{\prime},\ell^{\prime})$,

[TABLE]

Proof. We notice that the observation $x_{ij}$ is in block $(h,\ell)$ if and only if row $i$ is in cluster $h$ and column $j$ is in cluster $\ell$. Thanks to this remark, we can easily see that an observation $x_{ij}$ belongs to block $(h,\ell)$ and block $(h^{\prime},\ell^{\prime})$ if and only if row $i$ belongs at the same time to cluster $h$ and cluster $h^{\prime}$, and column $j$ belongs at the same time to cluster $\ell$ and cluster $\ell^{\prime}$.

∎

With the previous results, we finally have:

[TABLE]

∎

A.2 Proof of Corollary 2.4

Corollary 2.4.

$\forall(p,q)\in(H\times L)\times(H^{\prime}\times L^{\prime})$, we have the following relations between the margins,

[TABLE] 2. 2.

The CARI associated with the contingency table $\boldsymbol{n^{zwzw^{\prime}}}$ defined as in Equation (3) remains symmetric, that is to say,

[TABLE]

Proof.

This assertion forms part of the known properties of the Kronecker product. 2. 2.

The proof of this result is the direct consequence of the following lemma:

Lemma A.2**.**

We have the following relation,

[TABLE]

where t denotes the transpose of a matrix, $(\boldsymbol{z},\boldsymbol{w})$ and $(\boldsymbol{z^{\prime}},\boldsymbol{w^{\prime}})$ are two co-clustering partitions, $\boldsymbol{n^{zwz^{\prime}w^{\prime}}}$ is the corresponding contingency table defined according to Theorem 2.3., $\boldsymbol{n^{zz^{\prime}}}$ and $\boldsymbol{n^{ww^{\prime}}}$ are defined in Section 2.2.

Proof. Thanks to the property of the Kronecker product with the transpose, we have,

[TABLE]

∎

∎

A.3 Proof of Proposition 2.5

Proposition 2.5. Given two co-clustering partitions, $(\boldsymbol{z},\boldsymbol{w})$ and $(\boldsymbol{z^{\prime}},\boldsymbol{w^{\prime}})$, we have:

[TABLE]

Proof. Given two co-clustering partitions, $(\boldsymbol{z},\boldsymbol{w})$ and $(\boldsymbol{z^{\prime}},\boldsymbol{w^{\prime}})$, and thanks to the results of Theorem 2.3, we observe that for all $(p,q)\in\{1,\ldots,HL\}\times\{1,\ldots,H^{\prime}L^{\prime}\}$, we have:

[TABLE]

where $(h_{p},\ell_{p})$ (resp. $(h^{\prime}_{q},\ell^{\prime}_{q})$) are the coordinates in the co-clustering partition $(\textbf{z},\textbf{w})$ (resp. $(\textbf{z}^{\prime},\textbf{w}^{\prime})$) associated with the block $p$ (resp. $q$). For the same reason and thanks to the results of Corollary 2.4, we have:

[TABLE]

With these findings, we have:

[TABLE]

∎

A.4 Proof of the complexities

Complexity of ENMI and coNMI.

For the complexity of ENMI, we explain the computational cost for $NMI(\textbf{z},\textbf{z}^{\prime})$ as follows:

The computation of $(P_{h,h^{\prime}})_{1\leq h\leq H,1\leq h^{\prime}\leq H^{\prime}}$: $\mathcal{O}\left(I\right)$ because we look for each row $i$ of the initial matrix to which cell $(h,h^{\prime})$ of the contingency table it belongs to and we add 1 to its value. 2. 2.

For the same reasons, the computation of $(P_{h})_{1\leq h\leq H}$ (resp. $(P_{h^{\prime}})_{1\leq h^{\prime}\leq H^{\prime}}$) is $\mathcal{O}\left(HI\right)$. 3. 3.

Finally, the computation of the criterion $NMI(\textbf{z},\textbf{z}^{\prime})$ given the previous matrices is a sum on each pair $(h,h^{\prime})\in\{1,\ldots,H\}\times\{1,\ldots,H^{\prime}\}$ so this step has a complexity of $\mathcal{O}\left(HH^{\prime}\right)$.

So, the complexity of the $NMI(\textbf{z},\textbf{z}^{\prime})$ computation is $\mathcal{O}\left(max(HH^{\prime},I)\right)$.

As there is a similar complexity for the computation of $NMI(\textbf{w},\textbf{w}^{\prime})$, we have the result.

Complexity of CARI.

For the complexity of CARI, we proceed in three steps:

As $\boldsymbol{n^{zz^{\prime}}}=(IP_{h,h^{\prime}})_{1\leq h\leq H,1\leq h^{\prime}\leq H^{\prime}}$, the complexity is $\mathcal{O}\left(I\right)$. It is the same reasoning for the computation of $\boldsymbol{n^{zwz^{\prime}w^{\prime}}}$. 2. 2.

Thanks to the equation 3, the computation of $\boldsymbol{n^{zwz^{\prime}w^{\prime}}}$ is only the Kronecker product of the matrices $\boldsymbol{n^{zz^{\prime}}}$ and $\boldsymbol{n^{ww^{\prime}}}$. The complexity is $\mathcal{O}\left(HH^{\prime}LL^{\prime}\right)$. 3. 3.

Given the matrices $\boldsymbol{n^{zz^{\prime}}}$, $\boldsymbol{n^{ww^{\prime}}}$ and $\boldsymbol{n^{zwz^{\prime}w^{\prime}}}$, the most time-consuming operation of the CARI’s computation is $\sum_{p,q}\binom{n_{p,q}^{zwz^{\prime}w^{\prime}}}{2}$ what has a complexity of $\mathcal{O}\left(HH^{\prime}LL^{\prime}\right)$.

Finally, the complexity is $\mathcal{O}\left(\max\left(HH^{\prime}LL^{\prime},I,J\right)\right)$.

Complexity of CE.

For the complexity of $CE$, we have:

For each permutation $\sigma\in\mathfrak{S}(\{1,...,H\})$, we compute:

• •

$\frac{1}{I}\sum_{i,h}z_{ih}z^{\prime}_{i\sigma(h)}$ with a complexity of $\mathcal{O}\left(HI\right)$.

As the cardinal number of $\mathfrak{S}(\{1,...,H\})$ is $H!$, we obtain a complexity of $\mathcal{O}\left(H!HI\right)$. 2. 2.

We have a similar result for the computation of $\text{dist}_{\scriptsize{J,L}}(\boldsymbol{w},\boldsymbol{w^{\prime}})$; that is to say, we have $\mathcal{O}\left(L!LJ\right)$. 3. 3.

Given $\text{dist}_{\scriptsize{I,H}}(\boldsymbol{z},\boldsymbol{z^{\prime}})$ and $\text{dist}_{\scriptsize{J,L}}(\boldsymbol{w},\boldsymbol{w^{\prime}})$, the computation of $CE$ is only three operations.

At the end, the complexity of $CE$ is $\mathcal{O}\left(max\left(H!HI,L!LJ\right)\right)$.

Complexity of NCE.

For the complexity of $NCE$, we use the hungarian algorithm with a quartic complexity:

The complexity to cumpute the contingency table $\boldsymbol{n^{zz^{\prime}}}$ is of the order of $\mathcal{O}\left(I\right)$ as stated previously. 2. 2.

optimizing the strongest diagonal using the Hungarian algorithm has a complexity of $\mathcal{O}\left(\max(H^{4},H^{\prime 4})\right)$.

Thus, computing the distance of the partitions z and $\textbf{z}^{\prime}$,has a complexity of $\mathcal{O}\left(\max(H^{4},H^{\prime 4},I)\right)$. We have the similar result for the partition on columns and the result of the global complexity is proved.

A.5 Proof of the boundaries

Boundaries of ENMI.

For the boundaries of ENMI, we can observe that $MI(\textbf{z},\textbf{z}^{\prime})$ is the Kullback-Leibler divergence between the empirical distribution of the joint distribution and the product of the empirical marginal distributions.

Lower bound.

In particular, we know that the Kullback-Leibler divergence is greater than [math]; obtained in the case where the empirical density $(P_{h,h^{\prime}})_{1\leq h\leq H,1\leq h^{\prime}\leq H^{\prime}}$ is the product of the two empirical marginal distributions:

[TABLE]

In this case, we obtain:

[TABLE]

The lower bound of $\text{EMMI}((\textbf{z},\textbf{w}),(\textbf{z}^{\prime},\textbf{w}^{\prime}))$ is a consequence of the sum of $\text{MI}(\textbf{z},\textbf{z}^{\prime})$ and $\text{MI}(\textbf{w},\textbf{w}^{\prime})$ whose are both positive.

Upper bound on MI.

At the opposite, the upper bound is obtained when z is a sub-partition of $\textbf{z}^{\prime}$ if $H>H^{\prime}$ (or inversely). In this case, for each $h\in\{1,\ldots,H\}$, there exists $h^{\prime}\in\{1,\ldots,H^{\prime}\}$ so that for all $i\in\{1,\ldots,I\}$, $z_{ih}=1$ implies that $z_{ih^{\prime}}=1$; in this case, we say that $h$ is in the subset $S_{h^{\prime}}$ and we have:

[TABLE]

and we have:

[TABLE]

Then, we have:

[TABLE]

The last equation is a consequence of the fact that z is a sub-partition of $\textbf{z}^{\prime}$.

As the minimum is less than the maximum, we observe that $\text{MI}(\textbf{z},\textbf{z}^{\prime})$ is smaller than $1$, the $\text{EMMI}((\textbf{z},\textbf{w}),(\textbf{z}^{\prime},\textbf{w}^{\prime}))$ is then smaller than $2$.

Boundaries of CARI.

To prove the upper bound of the lower bound, we use the case where the partitions z, $\textbf{z}^{\prime}$, w and $\textbf{w}^{\prime}$ are equidistributed and the intersections of a cluster of z (resp. w) with each cluster of $\textbf{z}^{\prime}$ (resp. $\textbf{w}^{\prime}$) have the same number of rows (resp. columns). In particular, we have:

[TABLE]

With this configuration, we have:

[TABLE]

For the conjecture, we have:

[TABLE]

Upper bound.

As for every partition z, $\textbf{z}^{\prime}$, w and $\textbf{w}^{\prime}$, we have:

[TABLE]

then the upper bound is $1$.

Boundaries of (N)CE.

Let’s start by noting that $\sum\limits_{i,h}z_{ih}z^{\prime}_{i\sigma(h)}$ is smaller than $I$ for each permutation $\sigma\in\mathfrak{S}(\{1,...,H\})$. In this demonstration, we assume that $H\geq H^{\prime}$ and if $H^{\prime}<H$ then we complete partition $\boldsymbol{z^{\prime}}$ with empty clusters such that the matrix $\boldsymbol{n}^{\boldsymbol{zz^{\prime}}}$ is square. Thanks to the formulation under an assignment problem, we have shown that minimizing the criterion in Equation (4) leads to seek the optimal permutation on the columns of the matrix $\boldsymbol{n}^{\boldsymbol{zz^{\prime}}}$ which would provide the diagonal with the highest sum. Given a permutation $\sigma\in\mathfrak{S}(\{1,...,H\})$, $\boldsymbol{n}^{\sigma}$ denotes the matrix $\boldsymbol{n}^{\sigma}=\left(\boldsymbol{n}_{h\sigma(h^{\prime})}^{\boldsymbol{zz^{\prime}}}\right)_{1\leq h,h^{\prime}\leq H}$ after permutation and $S_{\sigma}=\sum\limits_{h=1}^{H}\boldsymbol{n}_{hh}^{\sigma}$ the sum of the cells on the diagonal $D_{\sigma}=\left(\boldsymbol{n}^{\sigma}_{11},\ldots,\boldsymbol{n}^{\sigma}_{HH}\right)$. Without loss of generality, we assume that the optimal permutation $\sigma$ is identity Id and we search the worst possible value for $S_{\textbf{Id}}$ which means that:

• •

The value of $S_{\textbf{Id}}$ must be as small as possible.

• •

For each permutation $\sigma\in\mathfrak{S}(\{1,...,H\})$, $S_{\textbf{Id}}\geq S_{\sigma}$.

If $\sigma_{h}$ denotes the circular permutation such that for all $h^{\prime}\in\{1,\ldots,H\}$,

[TABLE]

the n-tuple $\left(D_{\textbf{Id}},D_{\sigma_{1}},\ldots,D_{\sigma_{H-1}}\right)$ is a partition of the cells of matrix $\boldsymbol{n}^{\boldsymbol{zz^{\prime}}}$. It means that,

[TABLE]

As a result, we have:

[TABLE]

that is to say $S_{\textbf{Id}}\geq I/H$ and the equality is reached if all the cells of matrix $\boldsymbol{n}^{\boldsymbol{zz^{\prime}}}$ are identical.

In the end, for any permutation $\sigma\in\mathfrak{S}(\{1,...,H\})$, we have:

[TABLE]

The last inequality enables us to conclude.

In the case where $I$ is not divisible by $H^{2}$, the bound can be improved by regularly distributing the numbers.