Spectral Clustering via Ensemble Deep Autoencoder Learning (SC-EDAE)

TL;DR

This paper introduces a robust spectral clustering method that combines deep autoencoder learning within an ensemble framework, improving clustering stability and performance over existing deep clustering techniques.

Contribution

The paper proposes a novel ensemble deep autoencoder approach integrated with spectral clustering to enhance robustness and accuracy in data clustering tasks.

Findings

Outperforms state-of-the-art deep clustering methods on benchmark datasets

Demonstrates increased robustness to hyperparameter settings

Shows improved clustering stability and accuracy

Abstract

Recently, a number of works have studied clustering strategies that combine classical clustering algorithms and deep learning methods. These approaches follow either a sequential way, where a deep representation is learned using a deep autoencoder before obtaining clusters with k-means, or a simultaneous way, where deep representation and clusters are learned jointly by optimizing a single objective function. Both strategies improve clustering performance, however the robustness of these approaches is impeded by several deep autoencoder setting issues, among which the weights initialization, the width and number of layers or the number of epochs. To alleviate the impact of such hyperparameters setting on the clustering performance, we propose a new model which combines the spectral clustering and deep autoencoder strengths in an ensemble learning framework. Extensive experiments on various benchmark datasets demonstrate the potential and robustness of our approach compared to state-of-the-art deep clustering methods.

Figures

Tables

Table 1. Table 1: Description of the used FCPS data sets.

Data sets	Characteristics
	Samples	Features	Clusters	Main Problem
Tetra	400	3	4	inner vs inter cluster distances
Chainlink	1000	3	2	not linear separable
Lsun	400	2	3	different variances

Table 2. Table 2: Class distribution for MNIST , PenDigits and USPS datasets.

	0	1	2	3	4	5	6	7	8	9
MNIST	5923	6742	5958	6131	5842	5421	5918	6265	5851	5949
PenDigits	780	779	780	719	780	720	720	778	719	719
USPS	1194	1005	731	658	652	556	664	645	542	644

Table 3. Table 3: Mean clustering accuracy for LSC and k-means on original real datasets and encodings: Evaluations on MNIST, PenDigits, USPS data and their encodings. Bold values highlight the higher accuracy values.

Data	LSC	kmeans++	DAE structure	DAE-LSC	DAE-kmeans++
MNIST	68.55 $\pm 2.25$	55.13 $\pm 0.05$	500–750–1000	87.06 $\pm 8.27$	76.33 $\pm 7.69$
			500–1000–750	90.48 $\pm 5.20$	79.22 $\pm 5.93$
			750–500–1000	88.31 $\pm 5.46$	77.71 $\pm 6.03$
			750–1000–500	90.30 $\pm 4.89$	79.45 $\pm 5.81$
			1000–500–750	91.54 $\pm 3.06$	79.98 $\pm 5.98$
			1000–750–500	90.96 $\pm 3.98$	77.70 $\pm 5.09$
PenDigits	80.17 $\pm 3.76$	73.89 $\pm 3.97$	500–750–1000	85.59 $\pm 2.34$	73.64 $\pm 4.00$
			500–1000–750	85.11 $\pm 3.15$	74.67 $\pm 3.43$
			750–500–1000	85.36 $\pm 2.91$	73.47 $\pm 3.89$
			750–1000–500	85.27 $\pm 2.92$	74.64 $\pm 4.01$
			1000–500–750	85.02 $\pm 2.72$	74.20 $\pm 3.84$
			1000–750–500	84.39 $\pm 3.04$	73.78 $\pm 3.55$
USPS	77.20 $\pm 1.49$	68.36 $\pm 0.08$	500–750–1000	81.78 $\pm 8.08$	72.85 $\pm 3.52$
			500–1000–750	83.47 $\pm 7.40$	73.44 $\pm 3.70$
			750–500–1000	79.72 $\pm 6.21$	72.46 $\pm 2.78$
			750–1000–500	80.29 $\pm 5.70$	73.80 $\pm 3.51$
			1000–500–750	81.39 $\pm 4.46$	74.07 $\pm 3.07$
			1000–750–500	83.08 $\pm 5.64$	72.41 $\pm 3.06$

Table 4. Table 4: Mean clustering accuracy for SC-EDAE , ensemble on initializations, epochs number and structures: Bold values highlight the higher accuracy values.

Dataset	DAE structure	Ens.Init.	Ens.Ep.	Ens.Struct.
MNIST	500–750–1000	89.19 $\pm 0.41$	85.54 $\pm 4.30$	93.23 $\pm 2.84$
	500–1000–750	95.33 $\pm 0.07$	94.34 $\pm 2.68$
	750–500–1000	92.15 $\pm 0.25$	92.03 $\pm 3.87$
	750–1000–500	92.65 $\pm 0.13$	92.26 $\pm 3.71$
	1000–500–750	94.28 $\pm 0.20$	94.57 $\pm 1.48$
	1000–750–500	93.87 $\pm 0.38$	95.25 $\pm 0.59$
PenDigits	$500$-$750$-$1000$	86.80 $\pm 0.74$	87.08 $\pm 1.10$	86.44 $\pm 1.42$
	500–1000–750	85.95 $\pm 0.73$	86.69 $\pm 1.33$
	750–500–1000	86.69 $\pm 0.87$	87.27 $\pm 0.60$
	750–1000–500	86.48 $\pm 1.09$	86.91 $\pm 1.01$
	1000–500–750	86.75 $\pm 6.40$	86.96 $\pm 8.10$
	1000–750–500	86.66 $\pm 9.50$	87.28 $\pm 0.48$
USPS	500–750–1000	80.07 $\pm 1.95$	81.36 $\pm 5.09$	81.78 $\pm 3.61$
	500–1000–750	80.54 $\pm 0.77$	82.06 $\pm 3.54$
	750–500–1000	79.49 $\pm 1.19$	81.10 $\pm 3.86$
	750–1000–500	79.29 $\pm 1.05$	79.88 $\pm 2.69$
	1000–500–750	84.12 $\pm 1.80$	81.89 $\pm 3.21$
	1000–750–500	85.22 $\pm 2.14$	84.96 $\pm 3.29$

Table 5. Table 5: Mean clustering accuracy for SC-EDAE , ensemble on landmarks: Bold values highlight the higher accuracy values .

DAE structure	MNIST	PenDigits	USPS
500–750–1000	88.84 $\pm 1.22$	87.31 $\pm 1.13$	82.17 $\pm 3.79$
500–1000–750	95.35 $\pm 0.20$	87.21 $\pm 0.36$	81.96 $\pm 2.74$
750–500–1000	92.48 $\pm 1.27$	87.16 $\pm 0.99$	80.61 $\pm 3.46$
750–1000–500	92.53 $\pm 0.76$	87.09 $\pm 0.95$	80.30 $\pm 1.26$
1000–500–750	93.76$\pm 1.14$	86.67 $\pm 1.40$	86.35 $\pm 2.62$
1000–750–500	95.08 $\pm 0.17$	87.13 $\pm 1.26$	87.32 $\pm 4.85$

Table 6. Table 6: Mean clustering accuracy and NMI comparison with deep k-means variants: Mean accuracy and NMI for MNIST and USPS over 10 10 10 replicates with SC-EDAE and comparison to baselines and state-of-the-art approaches. Bold values highlight the higher accuracy values .

Model	MNIST		USPS
	ACC	NMI	ACC	NMI
baselines
kmeans++	55.13 $\pm 0.05$	52.89 $\pm 0.02$	68.36 $\pm 0.08$	65.67 $\pm 0.10$
LSC	68.55 $\pm 2.25$	70.54 $\pm 0.83$	77.20 $\pm 1.49$	79.48 $\pm 0.90$
DAE+kmeans++	78.40 $\pm 6.09$	71.97 $\pm 4.13$	73.17 $\pm 3.27$	70.48 $\pm 1.84$
DAE+LSC	89.78 $\pm 5.14$	83.06 $\pm 4.38$	81.62 $\pm 6.25$	80.44 $\pm 3.39$
no pretraining required
SC-EDAE Ens.Init.	92.91 $\pm 0.24$	87.65 $\pm 0.18$	81.46 $\pm 1.48$	82.88 $\pm 0.59$
SC-EDAE Ens.Ep.	92.33 $\pm 2.77$	87.72 $\pm 2.42$	81.88 $\pm 3.62$	83.03 $\pm 1.88$
SC-EDAE Ens.Struct.	93.23 $\pm 2.84$	87.93 $\pm 2.27$	81.78 $\pm 3.61$	83.17 $\pm 1.96$
Deep clustering approaches without pretraining (Fard et al. 2018) [46]
DCNnp	34.8 $\pm 3.0$	18.1 $\pm 1.0$	36.4 $\pm 3.5$	16.9 $\pm 1.3$
IDECnp	61.8 $\pm 3.0$	62.2 $\pm 1.6$	53.9 $\pm 5.1$	50.0 $\pm 3.8$
DKMa	82.3 $\pm 3.2$	78.0 $\pm 1.9$	75.5 $\pm 6.8$	73.0 $\pm 2.3$
Deep clustering approaches with pretraining (Fard et al. 2018) [46]
DCNp	81.1 $\pm 1.9$	75.7 $\pm 1.1$	73.0 $\pm 0.8$	71.9 $\pm 1.2$
IDECp	85.7 $\pm 2.4$	86.4 $\pm 1.0$	75.2 $\pm 0.5$	74.9 $\pm 0.6$
DKMp	84.0 $\pm 2.2$	79.6 $\pm 0.9$	75.7 $\pm 1.3$	77.6 $\pm 1.1$

Table 7. Table 7: Mean clustering Adjusted Rand Index (ARI) and Normalized Mutual Information (NMI) for LSC and k-means on original real datasets and encodings. Evaluations on MNIST, PenDigits, USPS data and their encodings. Bold values highlight the higher results

	ARI		NMI			ARI		NMI
Data	LSC	kmeans++	LSC	kmeans++	DAE structure	DAE-LSC	DAE-kmeans++	DAE-LSC	DAE-kmeans++
MNIST	54.86 $\pm 1.69$	39.98 $\pm 0.03$	70.54 $\pm 0.83$	52.89 $\pm 0.02$	500–750–1000	78.16 $\pm 10.26$	63.58 $\pm 8.14$	80.88 $\pm 6.58$	70.29 $\pm 5.38$
					500–1000–750	82.84 $\pm 1.20$	67.66 $\pm 6.36$	84.04 $\pm 1.20$	73.21 $\pm 4.02$
					750–500–1000	79.20 $\pm 8.42$	65.32 $\pm 7.36$	81.57 $\pm 5.55$	71.50 $\pm 4.64$
					750–1000–500	82.23 $\pm 6.33$	66.75 $\pm 6.48$	83.52 $\pm 4.13$	72.32 $\pm 4.10$
					1000–500–750	83.66 $\pm 4.23$	67.48 $\pm 5.80$	84.29 $\pm 2.80$	72.80 $\pm 3.52$
					1000–750–500	83.15 $\pm 4.81$	65.28 $\pm 7.48$	84.07 $\pm 2.99$	71.69 $\pm 3.09$
PenDigits	68.58 $\pm 3.79$	57.58 $\pm 2.61$	79.78 $\pm 1.42$	69.72 $\pm 0.58$	500–750–1000	74.12 $\pm 2.53$	59.62 $\pm 3.79$	81.06 $\pm 1.43$	69.33 $\pm 2.11$
					500–1000–750	73.18 $\pm 3.55$	58.97 $\pm 3.41$	80.46 $\pm 1.85$	69.14 $\pm 1.99$
					750–500–1000	73.47 $\pm 3.12$	58.23 $\pm 3.73$	80.55 $\pm 1.48$	68.56 $\pm 2.25$
					750–1000–500	73.30 $\pm 3.17$	58.82 $\pm 3.73$	80.38 $\pm 1.62$	68.74 $\pm 2.07$
					1000–500–750	73.07 $\pm 2.97$	58.92 $\pm 3.35$	80.23 $\pm 1.79$	69.53 $\pm 2.23$
					1000–750–500	73.40 $\pm 3.17$	58.16 $\pm 3.12$	80.66 $\pm 1.60$	68.83 $\pm 1.90$
USPS	77.09 $\pm 1.52$	57.70 $\pm 0.12$	79.48 $\pm 0.90$	65.67 $\pm 0.10$	500–750–1000	76.12 $\pm 8.45$	63.62 $\pm 3.02$	80.32 $\pm 4.89$	70.35 $\pm 2.25$
					500–1000–750	77.34 $\pm 7.71$	64.22 $\pm 3.34$	80.69 $\pm 4.30$	70.37 $\pm 2.16$
					750–500–1000	73.66 $\pm 6.38$	63.34 $\pm 2.67$	78.77 $\pm 3.81$	70.11 $\pm 1.77$
					750–1000–500	75.17 $\pm 5.23$	64.87 $\pm 2.66$	80.13 $\pm 3.11$	70.94 $\pm 2.03$
					1000–500–750	76.15 $\pm 4.29$	64.63 $\pm 2.02$	80.98 $\pm 2.12$	70.80 $\pm 1.36$
					1000–750–500	77.75 $\pm 5.02$	63.88 $\pm 2.07$	81.74 $\pm 2.12$	70.33 $\pm 1.45$

Table 8. Table 8: Mean clustering Adjusted Rank Index (ARI) and Normalized Mutual Information (NMI) for the SC-EDAE algorithm. The ensemble is done on initializations, epochs number and structures. Bold values highlight the higher results.

		ARI			NMI
Data	DAE structure	Ens.Init.	Ens.Ep.	Ens.Struct.	Ens.Init.	Ens.Ep.	Ens.Struct.
MNIST	500–750–1000	81.87 $\pm 0.49$	83.22 $\pm 7.07$	87.25 $\pm 3.88$	84.69 $\pm 0.28$	85.44 $\pm 4.22$	87.93 $\pm 2.27$
	500–1000–750	90.17 $\pm 0.14$	88.84 $\pm 3.93$		89.59 $\pm 0.10$	88.87 $\pm 2.32$
	750–500–1000	84.66 $\pm 1.71$	85.29 $\pm 5.18$		86.86 $\pm 0.21$	86.68 $\pm 3.06$
	750–1000–500	86.18 $\pm 0.20$	85.86 $\pm 4.89$		87.44 $\pm 0.13$	87.17 $\pm 2.66$
	1000–500–750	88.47 $\pm 0.27$	88.86 $\pm 2.49$		88.53 $\pm 0.17$	88.71 $\pm 1.45$
	1000–750–500	88.59 $\pm 0.38$	90.02 $\pm 1.13$		88.81 $\pm 0.18$	89.44 $\pm 0.81$
PenDigits	500–750–1000	75.67 $\pm 0.80$	76.15 $\pm 1.23$	74.88 $\pm 1.57$	82.33 $\pm 0.48$	82.73 $\pm 0.78$	81.87 $\pm 0.84$
	500–1000–750	74.00 $\pm 0.88$	74.83 $\pm 1.82$		80.96 $\pm 0.43$	81.68 $\pm 0.99$
	750–500–1000	75.13 $\pm 0.95$	75.71 $\pm 0.81$		81.89 $\pm 0.48$	82.19 $\pm 0.51$
	750–1000–500	74.98 $\pm 1.20$	75.38 $\pm 1.18$		81.88 $\pm 0.61$	81.99 $\pm 0.73$
	1000–500–750	75.07 $\pm 0.69$	75.63 $\pm 0.89$		81.86 $\pm 0.39$	82.14 $\pm 0.70$
	1000–750–500	75.16 $\pm 0.10$	75.60 $\pm 0.76$		81.97 $\pm 0.52$	82.13 $\pm 0.57$
USPS	500–750–1000	75.68 $\pm 1.80$	76.85 $\pm 5.37$	77.61 $\pm 3.69$	81.93 $\pm 0.81$	82.39 $\pm 3.04$	83.17 $\pm 1.96$
	500–1000–750	76.12 $\pm 0.71$	77.67 $\pm 3.63$		82.29 $\pm 0.39$	83.03 $\pm 1.93$
	750–500–1000	74.32 $\pm 1.02$	76.53 $\pm 3.74$		81.69 $\pm 0.48$	82.07 $\pm 1.87$
	750–1000–500	75.33 $\pm 0.93$	75.70 $\pm 2.82$		82.34 $\pm 0.43$	82.35 $\pm 1.59$
	1000–500–750	79.93 $\pm 1.64$	77.73 $\pm 3.00$		84.28 $\pm 0.65$	83.58 $\pm 1.37$
	1000–750–500	80.96 $\pm 1.99$	80.79 $\pm 3.21$		84.75 $\pm 0.78$	84.75 $\pm 1.47$

Equations

$$\underset{\mathbf{B}\in {\mathbb{R}}^{n\times k}}{\max }Tr({\mathbf{B}}^{\top }\mathbf{SB})s.t.{\mathbf{B}}^{\top }\mathbf{B}=\mathbf{I}$$

$${\mathcal{J}}_{AE}(\theta ,\psi )=\sum _{i=1}^n\mathcal{L}({\mathbf{x}}_i,g_{\psi }(f_{\theta }({\mathbf{x}}_i))).$$

$$||\mathbf{X}-g_{{\psi }_{\ell }}\circ f_{{\theta }_{\ell }}(\mathbf{X})|{|}^2$$

$$z_{ij}^{\ell }=\Phi ({\mathbf{y}}_i^{\ell })=\frac{\mathcal{K}({\mathbf{y}}_i^{\ell },{\mathbf{u}}_j^{\ell })}{{\sum }_{j^{\prime }\in N_{(i)}}\mathcal{K}({\mathbf{y}}_i^{\ell },{\mathbf{u}}_{j^{\prime }}^{\ell })}$$

$${\mathbf{S}}_{\ell }\in {\mathbb{R}}^{n\times n},{\mathbf{S}}_{\ell }={\mathbf{Z}}_{\ell }{\Sigma }^{-1}{\mathbf{Z}}_{\ell }^{\top }\text{mbox}where\Sigma =diag({\mathbf{Z}}_{\ell }^{\top }\text{mathbbm}1).$$

$${\mathbf{S}}_{\ell }={\hat{\mathbf{Z}}}_{\ell }{\hat{\mathbf{Z}}}_{\ell }^{\top }\text{mbox}where{\hat{\mathbf{Z}}}_{\ell }={\mathbf{Z}}_{\ell }{\Sigma }^{-1/2}.$$

$$\bar{\mathbf{S}}=\frac{1}{m}\sum _{\ell =1}^m{\mathbf{S}}_{\ell }.$$

$$\underset{\mathbf{B}}{\max }Tr({\mathbf{B}}^{\top }\bar{\mathbf{S}}\mathbf{B})s.t.{\mathbf{B}}^{\top }\mathbf{B}=\mathbf{I}.$$

$$\bar{\mathbf{Z}}=\frac{1}{\sqrt{m}}[{\hat{\mathbf{Z}}}_1|\dots |{\hat{\mathbf{Z}}}_j|\dots |{\hat{\mathbf{Z}}}_m].$$

$$\underset{{\mathbf{B}}^{\top }\mathbf{B}=\mathbf{I}}{\max }Tr({\mathbf{B}}^{\top }\bar{\mathbf{S}}\mathbf{B})\iff \underset{{\mathbf{B}}^{\top }\mathbf{B}=\mathbf{I},\mathbf{M}}{\min }||\bar{\mathbf{Z}}-\mathbf{B}{\mathbf{M}}^{\top }|{|}_F^2.$$

$$\underset{{\mathbf{B}}^{\top }\mathbf{B}=\mathbf{I},\mathbf{M}}{\min }||\bar{\mathbf{Z}}-\mathbf{B}{\mathbf{M}}^{\top }|{|}_F^2$$

$$\bar{\mathbf{S}}=\bar{\mathbf{Z}}{\bar{\mathbf{Z}}}^{\top }$$

$${\mathbf{x}}_i=\sigma (\mathbf{U}\sigma (\mathbf{W}{\mathbf{h}}_i))$$

Full text

††journal: Journal of LaTeX Templates

1 Introduction

Learning from large amount of data is a very challenging task. Several dimensionality reduction and clustering techniques that are well studied in the literature aim to learn a suitable and simplified data representation from original dataset; see for instance [1, 2, 3]. While many approaches have been proposed to address the dimensionality reduction and clustering tasks, deep learning-based methods recently demonstrate promising results. Motivated by the keen interest in deep learning, many authors tackle the objective of data representation and partitioning using jointly the autoencoders [4] and clustering approaches.

1.1 Deep Autoencoder: challenges and issues

Deep learning is a machine learning method that works with multi-level learning of data representations [5] where one passes from low level features to higher level features through the different layers. These deep architectures can automatically learn important features from images, sound or text data and have made significant progress in the field of computer vision. The autoencoder (AE) algorithm and its deep version (DAE), like the traditional methods of dimensionality reduction, has been a great success in recent years.

An autoencoder [4, 6, 7] is a neural network which is trained to replicate its input at its output. Training an autoencoder is unsupervised in the sense that no labeled data is needed. The training process is still based on the optimization of a cost function. Autoencoders can be used as tools to train deep neural networks [8].

For the purpose of dimensionality reduction, an autoencoder can learn a representation (or encoding) for a set of data. If linear activations are used, or only a single sigmoid hidden layer, then the optimal solution to an autoencoder is strongly related to Principal Component Analysis (PCA). With appropriate dimensionality and sparsity constraints, autoencoders can learn data projections that are more interesting than other basic techniques such as PCA which only allows linear transformation of data vectors. By contrast, the autoencoders are non-linear by nature, and can learn more complex relations between visible and hidden units. Moreover, they can be stacked, which makes them even more powerful.

Recently, a number of works have studied clustering strategies that combine classical clustering algorithms and deep learning methods. These approaches follow either a sequential way, where a deep representation is learned using a deep autoencoder before obtaining clusters using a clustering technique (e.g. k-means)  [9, 10, 11, 12, 13, 14, 15, 16, 17], or a simultaneous way, where deep representation and clusters are learned jointly by optimizing a single objective function [18, 19, 20]. Both strategies improve clustering performance. However, when dealing with real-world data, existing clustering algorithms based on deep autoencoders suffer from different issues which impede their robustness and ease-to-use, such as,

the weights initialization, as mentioned in [21], the training of a Deep Neural Network (DNN) still suffers from two major drawbacks, among which the weights initialization. Indeed, initializing the weights with random values clearly adds randomness to the obtained results. The DNN pretraining [22], which is strongly related to the initialization issue, has been used in an increasing number of studies [18, 23, 24]. While pretraining helps to improve clustering performance, it is usually computationally intensive and thus raises supplementary training issues.

• 2.

the architecture (or structure), the architecture (i.e., number of layers and their width) forces the network to seek a different representation of the data while preserving the important information. However, we observe that in almost all recent papers on deep clustering [18, 19, 20, 15, 16, 17, 25], a different structure is recommended by the authors for each studied dataset. In some studies, the DAE architecture can even lack of technical rationales. Most importantly, the clustering performance of the proposed methods usually strongly depends on a particular DAE structure.

1.2 Our paper’s contribution and structure

To address the above mentioned challenging issues, we propose a Spectral Clustering via Ensemble Deep Autoencoder’s algorithm (SC-EDAE) which combines the advantages and strengths of spectral clustering, deep embedding models and ensemble paradigm. Ensemble learning has been considered in different machine learning context where it generally helps in improving results by combining several models. The ensemble approach allows a better predictive performance and a more robust clustering as compared to the results obtained with a single model. Following the ensemble paradigm, we first used several DAE with different hyperparameters settings to generate $m$ encodings. In a second step, each encoding is projected in a higher features space based on the anchors strategy [26, 27] to construct $m$ graph affinity matrices. Finally, we apply spectral clustering on an ensemble graph affinity matrix to have the common space shared by all the $m$ encodings, before we run k-means in this common subspace to produce the final clustering (see Fig. 1 for a summary diagram).

The outline of the paper is as follows. In Section 2 we present the related work. In Section 3, some notations and preliminaries are given. In Section 4, we present and discuss our approach in full details. In Section 5, the evaluations of the proposed method and comparisons with several related approaches available in the literature are presented. The conclusion of the paper is given in Section 6.

2 Related Work

Despite their success, most existing clustering methods are severely challenged by the data generated with modern applications, which are typically high-dimensional, noisy, heterogeneous and sparse. This has driven many researchers to investigate new clustering models to overcome these difficulties. One promising category of such models relies on data embedding.

Within this framework, classical dimensionality reduction approaches, e.g., Principal Component Analysis (PCA), have been widely considered for the embedding task. However, the linear nature of such techniques makes it challenging to infer faithful representations of real-world data, which typically lie on highly non-linear manifolds. This motivates the investigation of deep learning models (e.g., autoencoders, convolutional neural networks), which have been shown so far to be successful in extracting highly non-linear features from complex data, such as text, images or graphs [4, 6, 7].

The deep autoencoders (DAE) have proven to be useful for dimensionality reduction [4] and image denoising. In particular, the autoencoders (AE) can non-linearly transform data into a latent space. When this latent space has lower dimension than the original one [4], this can be viewed as a form of non-linear PCA. An autoencoder typically consists of an encoder stage, that can provide an encoding of the original data in lower dimension, and a decoder part, to define the data reconstruction cost. In clustering context, the general idea is to embed the data into a low dimensional latent space and then perform clustering in this new space. The goal of the embedding here is to learn new representations of the objects of interest (e.g., images) that encode only the most relevant information characterizing the original data, which would for example reduce noise and sparsity.

Several interesting works have recently combined embedding learning and clustering. The proposed methods generally conduct both clustering and deep embedding in two different ways. First, some works proposed to combine deep embedding and clustering in a sequential way. In [10] the authors use a stacked autoencoder to learn a representation of the affinity graph, and then run k-means on the learned representations to obtain the clusters. In [24], it has been proposed to train a deep network by iteratively minimizing a Kullback-Leibler (KL) divergence between a centroid based probability distribution and an auxiliary target distribution.

More recently, in [28] the authors propose to incorporate an autoencoder into the Deep Embedded Clustering (DEC) framework [24]. Then, the proposed framework can jointly perform clustering and learn representative features with local structure preservation. A novel non-linear reconstruction method which adopt deep neural networks for representation based community detection has been proposed in [20]. The work presented in [25] combines deep learning with subspace clustering such that the network is designed to directly learn the affinities matrix. Finally, a novel algorithm was introduced in [15] that uses landmarks and deep autoencoders, to perform efficient spectral clustering.

Since the embedding process is not guaranteed to infer representations that are suitable for the clustering task, several authors recommend to perform both tasks jointly so as to let clustering govern feature extraction and vice-versa. In [19], the authors propose a general framework, so-called DeepCluster, to integrate the traditional clustering methods into deep learning models and adopt Alternating Direction of Multiplier Method to optimize it. In [18], a joint dimensionality reduction and k-means clustering approach in which dimensionality reduction is accomplished via learning a deep neural network is proposed.

Beyond the joint and sequential ways to combine clustering and deep embedding, it appears that the connection between autoencoder and ensemble learning paradigm has not been explored yet. In this paper, we aim to fill the gap between ensemble deep autoencoders and spectral clustering in order to propose a robust approach that takes simultaneously advantage of several deep models with various hyperparameter settings. In particular, we apply spectral clustering on an ensemble of fused encodings obtained from $m$ different deep autoencoders. To our knowledge, the adoption of deep learning in an ensemble learning paradigm has not been adequately investigated yet. The goal of this work is to conduct investigations along this direction.

3 Preliminaries

3.1 Notation

Throughout the paper, we use bold uppercase characters to denote matrices, bold lowercase characters to denote vectors. For any matrix $\mathbf{M}$, $\mathbf{m}_{j}$ denotes the $j$-th column vector of $\mathbf{M}$, $\mathbf{y}_{i}$ means the $i$-th row vector of $\mathbf{Y}$, $m_{ij}$ denotes the $(i,j)-$ element of $\mathbf{M}$ and $Tr[\mathbf{M}]$ is the trace of $\mathbf{M}$ whether $\mathbf{M}$ is a square matrix; $\mathbf{M}^{\top}$ denotes the transpose matrix of $\mathbf{M}$. We consider the Frobenius norm of a matrix $\mathbf{M}\in\mathbb{R}^{n\times d}$: $||\mathbf{M}||^{2}=\sum_{i=1}^{n}\sum_{j=1}^{d}m_{ij}^{2}=Tr[\mathbf{M}^{\top}\mathbf{M}]$. Furthermore, let $\mathbf{I}$ be the identity matrix with appropriate size.

3.2 Spectral clustering

Spectral clustering is a popular clustering method that uses eigenvectors of a symmetric matrix derived from the distance between datapoints. Several algorithms have been proposed in the literature [29, 30], each using the eigenvectors in slightly different ways [31, 32, 33]. The partition of the $n$ datapoints of $\mathbf{X}~{}\in\mathbb{R}^{n\times d}$ into $k$ disjoint clusters is based on an objective function that favors low similarity between clusters and high similarity within clusters. In its normalized version, the spectral clustering algorithm exploits the top $k$ eigenvectors of the normalized graph Laplacian $\mathbf{L}$ that are the relaxations of the indicator vectors which provide assignments of each datapoint to a cluster. In particular, it amounts to maximize the following relaxed normalized association,

[TABLE]

with $\mathbf{S}=\mathbf{D}^{-1/2}\mathbf{K}\mathbf{D}^{-1/2}~{}\in\mathbb{R}^{n\times n}$ is the normalized similarity matrix where $\mathbf{K}~{}\in\mathbb{R}^{n\times n}$ is the similarity matrix and $~{}D\in\mathbb{R}^{n\times n}$ is the diagonal matrix whose $(i,i)$-element of $\mathbf{X}$ is the sum of $\mathbf{X}$’s $i$-th row. The solution of (1) is to set the matrix $\mathbf{B}~{}\in\mathbb{R}^{n\times k}$ equal to the $k$ eigenvectors corresponding to the largest $k$ eigenvalues of $\mathbf{S}$. After renormalization of each row of $\mathbf{B}$, a k-means assigns each datapoint $\mathbf{x}_{i}$ of $\mathbf{X}$ to the cluster that the row $\mathbf{b}_{i}$ of $\mathbf{B}$ is assigned to.

As opposed to several other clustering algorithms (e.g. k-means), spectral clustering performs well on arbitrary shaped clusters. However, a limitation of this method is the difficulty to handle large-scale datasets due to the high complexity of the graph Laplacian construction and the eigendecomposition.

Recently, a scalable spectral clustering approach, referred to as Landmark-based Spectral Clustering (LSC) [34] or AnchorGraph [26], has been proposed. This approach allows to efficiently construct the graph Laplacian and compute the eigendecomposition. Specifically, each datapoint is represented by a linear combination of $p$ representative datapoints (or landmarks), with $p\ll n$. The obtained representation matrix $\hat{\mathbf{Z}}\in\mathbb{R}^{p\times n}$, for which the affinity is calculated between $n$ datapoints and the $p$ landmarks, is sparse which in turn ensures a more efficient eigendecomposition as compare to the above mentioned eigendecomposition of $\mathbf{S}$ (Eq. 1).

3.3 Deep autoencoders

An autoencoder [35] is a neural network that implements an unsupervised learning algorithm in which the parameters are learned in such a way that the output values tend to copy the input training sample. The internal hidden layer of an autoencoder can be used to represent the input in a lower dimensional space by capturing the most salient features.

Specifically, we can decompose an autoencoder in two parts, namely an encoder, $f_{\theta}$, followed by a decoder, $g_{\psi}$. The first part allows the computation of a feature vector $\mathbf{y}_{i}=f_{\theta}(\mathbf{x}_{i})$ for each input training sample, thus providing the encoding $\mathbf{Y}$ of the input dataset. The decoder part aims at transforming back the encoding into its original representation, $\hat{\mathbf{x}}_{i}=g_{\psi}(\mathbf{y}_{i})$.

The sets of parameters for the encoder $f_{\theta}$ and the decoder $g_{\psi}$ are learned simultaneously during the reconstruction task while minimizing the loss, referred to as $\mathcal{J}$, where $\cal{L}$ is a cost function for measuring the divergence between the input training sample and the reconstructed data,

[TABLE]

The encoder and decoder parts can have several shallow layers, yielding a deep autoencoder (DAE) that enables to learn higher order features. The network architecture of these two parts usually mirrors each other.

It is remarkable that PCA can be interpreted as a linear AE with a single layer [4]. In particular, PCA can be seen as a linear autoencoder with $\mathbf{W}\in\mathbb{R}^{d\times k}$ where $k\leq d$. Taking $f_{\theta}(\mathbf{X})=\mathbf{X}\mathbf{W}$ and $g_{\psi}\circ f_{\theta}(\mathbf{X})=\mathbf{X}\mathbf{W}\mathbf{W}^{\top}$ we find the objective function $||\mathbf{X}-\mathbf{X}\mathbf{W}\mathbf{W}^{\top}||^{2}$ optimized by PCA.

4 Spectral Clustering via Ensemble DAE

4.1 Problem formulation

Given an $n\times d$ data matrix $\mathbf{X}$, the goal is to first obtain a set of $m$ encodings $\{\mathbf{Y}_{\ell}\}_{\ell\in[1,m]}$ using $m$ DAE trained with different hyperparameters settings. In a second step, we construct a graph matrix $\mathbf{S}_{\ell}$ associated to each embedding $\mathbf{Y}_{\ell}$, and then fuse the $m$ graph matrices in an ensemble graph matrix $\mathbf{S}$ which contains information provided by the $m$ embeddings. Finally, to benefit from the common subspace shared by the $m$ deep embeddings, spectral clustering is applied to $\mathbf{S}$. The challenges of the problem are threefold,

generate $m$ deep embeddings, 2. 2.

integrate the clustering in an ensemble learning framework, 3. 3.

solve the clustering task in a highly efficient way.

Each of the above mentioned issues is discussed in the separate subsections 4.2, 4.3 and 4.4 respectively. Most importantly, the SC-EDAE approach is provided with an ensemble optimization which is detailed in subsection 4.5.

4.2 Deep embeddings generation

The cost function of an autoencoder, with an encoder $f_{\theta}$ and a decoder $g_{\psi}$, measures the error between the input $\mathbf{x}\in\mathbb{R}^{d\times 1}$ and its reconstruction at the output $\hat{\mathbf{x}}\in\mathbb{R}^{d\times 1}$. The encoder $f_{\theta}$ and decoder $g_{\psi}$ can have multiple layers of different widths. To generate $m$ deep representations or encodings $\{\mathbf{Y}_{\ell}\}_{\ell\in[1,m]}$, the DAE is trained with different hyperparameter settings (e.g., initialization, layer widths) by optimizing the following cost function.

[TABLE]

where $g_{\psi_{\ell}}$ and $f_{\theta_{\ell}}$ are learned with the hyperparameter setting $\ell$, and $\mathbf{Y}_{\ell}=f_{\theta_{\ell}}(\mathbf{X})$ (Fig. 1, $(a)$).

4.3 Graph matrix construction

To construct the graph matrix $\mathbf{S}_{\ell}$, we use an idea similar to that of Landmark Spectral Clustering [27] and the Anchor-Graphs [26], where a smaller and sparser representation matrix $\mathbf{Z}_{\ell}\in\mathbb{R}^{n\times p}$ that approximates a full $n\times n$ affinity matrix is built between the landmarks $\{\mathbf{u}_{j}^{\ell}\}_{j\in[1,p]}$ and the encoded points $\{\mathbf{y}_{i}^{\ell}\}_{i\in[1,n]}$ (Fig. 1, $(a)$). Specifically, a set of $p$ points ($p\ll n$) are obtained through a $k$-means clustering on the embedding matrix $\mathbf{Y}_{\ell}$. These points are the landmarks which approximate the neighborhood structure. Then a non-linear mapping from data to landmark is computed as follows,

[TABLE]

where $N_{(i)}$ indicates the $r$ ($r<p$) nearest landmarks around $\mathbf{y}_{i}^{\ell}$. As proposed in [27], we set $z_{ij}^{\ell}$ to zero when the landmark $\mathbf{u}_{j}^{\ell}$ is not among the nearest neighbor of $\mathbf{y}_{i}^{\ell}$, leading to a sparse affinity matrix $\mathbf{Z}_{\ell}$. The function $\mathcal{K}(.)$ is used to measure the similarity between data $\mathbf{y}_{i}^{\ell}$ and anchor $\mathbf{u}_{j}^{\ell}$ with $L_{2}$ distance in Gaussian kernel space $\mathcal{K}(\mathbf{x}_{i},\mathbf{x}_{j})=\exp(-||\mathbf{x}_{i}-\mathbf{x}_{j}||^{2}/2\sigma^{2})$, and $\sigma$ is the bandwidth parameter. The normalized matrix $\hat{\mathbf{Z}}_{\ell}\in\mathbb{R}^{n\times p}$ is then utilized to obtain a low-rank graph matrix,

[TABLE]

As the $\Sigma^{-1}$ normalizes the constructed matrix, $\mathbf{S}_{\ell}$ is bi-stochastic, i.e. the summation of each column and row equal to one, and the graph Laplacian becomes,

[TABLE]

4.4 Ensemble of affinity matrices

Given a set of $m$ encodings $\{\mathbf{Y}_{\ell}\}_{\ell\in[1,m]}$ obtained using $m$ DAE trained with different hyperparameters setting $\ell$, the goal is to merge the $m$ graph similarity matrices $\mathbf{S}_{\ell}$ in an ensemble similarity matrix which contains information provided by the $m$ embeddings. To aggregate the different similarity matrices, we use an Ensemble Clustering idea analogous to that proposed in [36, 37] where a co-association matrix is first built as the summation of all basic similarity matrices, and where each basic partition matrix can be represented as a block diagonal matrix. Thus, the SC-EDAE ensemble affinity matrix is built as the summation of the $m$ basics similarity matrices using the following formula,

[TABLE]

Note that the obtained matrix $\bar{\mathbf{S}}$ is bi-stochastic, as ${\mathbf{S}}_{\ell}$ (Eq. 6). For many natural problems, $\bar{\mathbf{S}}$ is approximately block stochastic matrix, and hence the first $k$ eigenvectors of $\bar{\mathbf{S}}$ are approximately piecewise constant over the $k$ almost invariant rows subsets [38].

In the sequel, we aim to compute, at lower cost, $\mathbf{B}$ that is shared by the $m$ graph matrices $\mathbf{S}_{\ell}$, and obtained by optimizing the following trace maximization problem

[TABLE]

4.5 Proposed optimization and algorithm

The solution of Eq. 7 is to set the matrix $\mathbf{B}$ equal to the $k$ eigenvectors corresponding to the largest $k$ eigenvalues of $\bar{\mathbf{S}}$. However, as the computation of the eigendecomposition of $\bar{\mathbf{S}}$ of size $(n\times n)$ is $O(n^{3})$, relying on proposition 4.1, we propose instead to compute the $k$ left singular vectors of the concatenated matrix,

[TABLE]

Using the sparse matrix $\bar{\mathbf{Z}}$ $\in\mathbb{R}^{n\times\sum_{j=1}^{m}\ell_{j}}$ with $\sum_{j=1}^{m}\ell_{j}\ll n$, instead of $\bar{\mathbf{S}}$, which has a larger dimension, naturally induces an improvement in the computational cost of $\mathbf{B}$ (Fig. 1, $(b)$).

Proposition 4.1**.**

Given a set of $m$ similarity matrices $\mathbf{S}_{\ell}$, such that each matrix $\mathbf{S}_{\ell}$ can be expressed as $\mathbf{Z}_{\ell}\mathbf{Z}_{\ell}^{\top}$. Let $\bar{\mathbf{Z}}\in\mathbb{R}^{n\times\sum_{j=1}^{m}\ell_{j}}$, where $\sum_{j=1}^{m}\ell_{j}\ll n$, denoted as $\frac{1}{\sqrt{m}}[\mathbf{Z}_{1}|\ldots|\mathbf{Z}_{j}|\ldots|\mathbf{Z}_{m}]$, be the concatenation of the $\mathbf{Z}_{\ell}$’s, $\ell=1,\ldots,m$. We first have,

[TABLE]

Then, given ${\tt SVD}(\bar{\mathbf{Z}})$, $\bar{\mathbf{Z}}=\mathbf{U}\Sigma\mathbf{V}^{\top}$ and the optimal solution $\mathbf{B}^{*}$ is equal to $\mathbf{U}$.

Proof.

From the second term of Eq. 9, one can easily show that $\mathbf{M}^{*}=\bar{\mathbf{Z}}^{\top}\mathbf{B}$. Plugging now the expression of $\mathbf{M}^{*}$ in Eq. 9, the following equivalences hold

[TABLE]

On the other hand, ${\tt SVD}(\bar{\mathbf{Z}})$ leads to $\bar{\mathbf{Z}}=\mathbf{U}\Sigma\mathbf{V}^{\top}$ (with $\mathbf{U}^{\top}\mathbf{U}=\mathbf{I}$, $\mathbf{V}^{\top}\mathbf{V}=\mathbf{I}$) and therefore to the eigendecomposition of $\bar{\mathbf{S}}$ as follows:

[TABLE]

Thereby the left singular vectors of $\bar{\mathbf{Z}}$ are the same as the eigenvectors of $\bar{\mathbf{S}}$. ∎

The steps of our SC-EDAE algorithm are summarized in Algorithm 1 and illustrated by Figure 1. The SC-EDAE approach proposes a unique way to combine DAE encodings with clustering. It also directly benefits from the low complexity of the anchors strategy for both the graph affinity matrix construction and the eigendecomposition.

Specifically, the computational cost for the construction of each $\mathbf{Z}_{\ell}$ affinity matrix amounts to $\mathcal{O}(np_{\ell}e(t+1))$ (Alg. 1, step (b)) , where $n$ is the number of datapoints, $p_{\ell}$ is the number of landmarks for the $\ell^{th}$ DAE ($p_{\ell}\ll n$), $e$ is the size of the DAE encoding $\mathbf{Y}_{\ell}$ ($e\ll n$) and $t$ is the number of iterations for the k-means that is used to select the landmarks. The computation of the $\mathbf{Z}_{\ell}$ matrices can be easily parallelized over multiple cores, leading to an efficient computation of the ensemble affinity matrix $\bar{\mathbf{Z}}$. Furthermore, the eigendecomposition of the sparse ensemble affinity matrix $\bar{\mathbf{Z}}$, which leads to the $\mathbf{B}$ embeddings (Alg. 1, step (c)), induces a computational complexity of $\mathcal{O}(p^{\prime 3}+p^{\prime}2n)$, where $p^{\prime}$ is the sum of all landmarks numbers for the concatenated $\mathbf{Z}_{\ell}$ matrices, i.e. $p^{\prime}=\sum_{j=1}^{m}\ell_{j}\ll n$. Finally, we need additional $\mathcal{O}(nctk)$ for the last k-means on $\mathbf{B}\in\mathbbm{R}^{n\times k}$ (Alg. 1, output), where $c$ is the number of centroïds, usually equal to $k$ the number of eigenvectors, leading to $\mathcal{O}(ntk^{2})$.

The originality and efficiency of our ensemble method hinges on the replacement of a costly eigendecomposition on $\bar{\mathbf{S}}\in\mathbbm{R}^{n\times n}$ by an eigendecomposition on a low-dimensional and sparse matrix $\mathbf{\bar{Z}}$ $\in\mathbb{R}^{n\times\sum_{j=1}^{m}\ell_{j}}$, with $\sum_{j=1}^{m}\ell_{j}\ll n$ (Alg. 1, step (c)). In particular, the sparsity of $\mathbf{\bar{Z}}$ enables the use of fast iterative and partial eigenvalue decomposition.

5 Experiments

5.1 Deep autoencoders settings

For our experiments, we trained fully connected autoencoders with an encoder $f_{\theta}$ of three hidden layers of size 50, 75 or 100 for synthetic datasets (Tetra, Chainlink and Lsun; Section 5.3), and three hidden layers of size 500, 750 or 1000 for real datasets (MNIST, PenDigits and USPS; Section 5.4), as suggested by Bengio et al. [8], in all possible orders. The decoder part $g_{\psi}$ mirrors the encoder stage $f_{\theta}$. For each DAE architecture (e.g., $\{750-500-1000\}$, $\{100-50-75\}$), 5 encodings were generated with 50, 100, 150, 200 and 250 epochs for real datasets and 200 epochs for synthetic datasets. The weights initialization follows the Glorot’s approach [39] and all encoder/decoder pairs used rectified linears units (ReLUs), except for the output layer which requires a sigmoid function. The autoencoder data are systematically $L_{2}$ normalized. We configure the autoencoders using the Keras tensorflow Python package, and compile the neural network with binary cross-entropy loss and Adam optimizer [40] with the default Keras parameters.

5.2 SC-EDAE ensemble strategy

The ensemble strategy of SC-EDAE exploits the encodings $\{\mathbf{Y}_{\ell}\}_{\ell\in[1,m]}$vwhich are generated with either (i) $m$ different DAE initializations or $m$ different DAE epochs number in association with one DAE structure (e.g. $d$–500–1000–750–$e$, with $d$ and $e$ the input and encoding layers width resp.), or (ii) $m$ DAE with different structures for the same number of landmarks and epochs. In both cases, the SC-EDAE strategy enables to compute the $m$ different sparse affinity matrices $\{\hat{\mathbf{Z}}_{\ell}\}_{\ell\in[1,m]}$ (Eq. 4) and, following Proposition 4.1, generate the ensemble affinity matrix $\mathbf{\bar{Z}}$ (Eq. 8).

5.3 Synthetic datasets

As a first step, we focus on synthetic datasets to illustrate the SC-EDAE algorithm and show the class-separability information embedded in the left singular vectors matrix of $\bar{\mathbf{Z}}$, noted as $\mathbf{B}^{\star}$ (Prop. 4.1 and Alg.1). We used generated synthetic data sets selected from the Fundamental Clustering Problem Suite (FCPS)111The suite can be downloaded from the website of the author: http://www.uni-marburg.de/fb12/datenbionik/data. FCPS yields some hard clustering problems, a short description of Tetra, Chainlink and Lsun FCPS data sets and the inherent problems related to clustering are given in Table 1. Following the experiments on synthetic data proposed by Yang et al. [18], we transformed the low-dimensional FCPS data, $\mathbf{h}_{i}\in\mathbb{R}^{2}$ or $\mathbb{R}^{3}$, in high-dimensional datapoints, $\boldsymbol{{x}_{i}}\in\mathbb{R}^{100}$. Specifically, the $\mathbf{x}_{i}$ are transformed based on the following equation,

[TABLE]

where the entries of matrices $\mathbf{W}\in\mathbb{R}^{10\times 2}$ and $\mathbf{U}\in\mathbb{R}^{100\times 10}$ follow the zero-mean unit-variance i.i.d. Gaussian distribution, and the sigmoid function $\sigma(.)$ introduces nonlinearity.

5.4 Real datasets

Our SC-EDAE algorithm (Alg.1) is fully evaluated on three image datasets, namely MNIST (Modified National Institute of Standards and Technology) [41], PenDigits (Pen-Based Recognition of Handwritten Digits) [42] and USPS (U.S. Postal Service) [43] and their DAE encodings (see Section 5.1 for details on DAE structure).

• MNIST [41] The database is loaded from the Keras Python package. The training and testing sets contain respectively $60,000$ and $10,000$ images of size $28\times 28$ of the integers in range $0-9$. The images are of grayscale levels rescaled within $[0,1]$ by dividing by $255$.
• PenDigits [42] The training and testing sets contain respectively $7,494$ and $3,498$ images of size $16\times 16$ of the integers in range $0-9$. The images with 16 numeric attributes rescaled within $[0,1]$ by dividing by $100$.
• USPS [43] The database is prepared as proposed in [23] and contains $9,298$ images of size $16\times 16$ pixels of the $10\mathchar 45\relax$ digits (integers in range $0-9$) rescaled within $[0,1]$.

The classes distribution for each dataset is given in Table 2. MNIST and PenDigits appear as balanced-class datasets while USPS has an imbalanced distribution.

5.5 Experiment results

5.5.1 Evaluation on synthetic data

Synthetic data enable us to easily explore the separability capacity of the embeddings matrix $\mathbf{B}$. For the experiments related to synthetic data, SC-EDAE is used in its ensemble structure version, with $m=6$ encodings from different structures, and the number of landmarks is set to 100. Applying SC-EDAE on the data sets Tetra, Chainlink and Lsun, we note that the 2D representations of the obtained clusters reflect the real cluster structure (Fig. 2 a, b, c; projection on the two first components of the matrix $\mathbf{B}$ as computed in Alg.1, step c). The SC-EDAE accuracy is of 1.00 for Tetra and Chainlink, and 0.90 for Lsun. The colored labels correspond to the predicted clusters. Complementary tests with different transformation functions confirm this trend (see annexes, Section A.1).

5.5.2 Baseline evaluations on real data

As baseline, we first evaluate k-means and LSC [27] on the three real datasets. The kmeans*++* approach corresponds to the scikit-learn Python package k-means implementation with the default parameters and kmeans*++* initialization scheme [44]. We implemented the LSC method in Python, following the Matlab implementation proposed in [27], and kept the same default parameters. The LSC landmarks initialization is done with k-means, which has been shown to provide better accuracy results than the random initialization [27, 15]. We consider landmarks number within $100$ and $1000$, by step of $100$. The evaluations are done either on the original datasets (Table 3, columns LSC and kmeans*++* or on the encodings (Table 3, columns DAE-LSC and DAE-kmeans++). The accuracy reported for LSC and k-means++ corresponds to the mean over $10$ clustering replicates on the original datasets, over all epoch and landmark numbers. The accuracy reported for DAE-LSC and DAE-kmeans++ corresponds to an average over 50 replicates (10 replicates on each of the 5 encodings per DAE structure), over all epoch and landmark numbers (see annexes for complementary results per DAE structure, Section A.2).

As can be seen from Table 3 and already reported in [27], LSC outperforms kmeans*++* for the clustering task on the three datasets (bold values, columns LSC and kmeans++), yet with larger standard deviations. The same trend is observed when applying LSC and kmeans++ on encodings, with standard deviations of similar magnitude for both clustering methods (bold values, columns DAE-LSC and DAE-kmeans++).

The results from Table 3 demonstrate that the simple combination of DAE and LSC or k-means already reaches higher accuracy and smaller standard deviations than without the autoencoder step. These results also show the advantage of associating the DAE encodings with the landmark-based representation over the k-means approach for the clustering task (columns DAE-LSC and DAE-kmeans++). In particular, the average accuracy for the MNIST and USPS datasets varies within $[87.06;91.54]$ and $[79.72;83.47]$ respectively for DAE-LSC and within $[77.70;79.98]$ and $[72.41;74.07]$ respectively for DAE-kmeans++.

Although the encodings generated by the deep autoencoder improve the clustering accuracy, finding a priori the most appropriate DAE structure remains a challenging task. The accuracy may also vary for different landmark and epoch numbers (see Table 5 and annexes Tables 7 & 8). As will be seen in the following sections, the ensemble strategy of SC-EDAE provides a straightforward way to alleviate these issues and avoid arbitrary DAE hyperparameters setting.

5.5.3 SC-EDAE ensemble evaluations

The Table 4 summarizes the performance of our LSC-based ensemble approach in the two cases detailed in section 5.2. Specifically, the columns Ens.Init. and Ens.Ep. indicate the clustering accuracy for the case (i) with an ensemble approach on the DAE weights initialization (Ens.Init., $m=5$) and the DAE training epoch numbers (Ens.Ep., $m=5$). The clustering accuracy values for the ensemble approach on various DAE structures, i.e. case (ii), is provided in the column Ens.Struct. ($m=6$).

The SC-EDAE ensemble strategy provides higher clustering accuracy as compare to the baseline evaluations (Table 3). In particular, the mean accuracy values obtained with the ensemble strategy for MNIST, PenDigits and USPS can reach, $95.33\pm 0.07$, $87.28\pm 0.48$ and $85.22\pm{2.14}$ respectively, vs. $91.54\pm 3.06$, $85.59\pm{2.34}$ and $83.47\pm{7.40}$ (Table 3).

The SC-EDAE ensemble approach on the DAE structures (Ens.Struct.) enables also to reach higher accuracy as compare to the baseline evaluations for MNIST ($93.23\pm 0.28$ vs. $91.54\pm 3.06$) and PenDigits ($86.44\pm 1.42$ vs. $85.59\pm{2.34}$), but with the added benefit of avoiding the arbitrary choice of a particular DAE structure. The SC-EDAE results for USPS with an ensemble on several structures are lower than our reference evaluations ($81.78\pm{3.61}$ vs. $83.47\pm{7.40}$), yet the accuracy value remains fairly high with lower standard deviation.

While the SC-EDAE method aims at providing an ensemble strategy for the deep architecture settings (Ens.Init., Ens.Ep. and Ens.Struct., Table 4), it relies also on the LSC idea which depends on the number of landmarks. We studied the possibility of an ensemble on the number of landmarks ($m=5$). As can be seen from Table 5, which provides mean accuracy on $10$ replicates, the ensemble strategy enables again to reach high accuracy values as compared to our baseline evaluations. The results still remain dependent from the DAE structure type, in particular for MNIST and USPS, and we would therefore recommend to use SC-EDAE in its ensemble structure version (ie., Ens.Struct.).

5.6 Evaluation in terms of NMI and ARI

Evaluating clustering results is not a trivial task. The clustering accuracy is not always a reliable measure when the clusters are not balanced and the number of clusters is high. To better appreciate the quality of our approach, in the sequel we retain two widely used measures to assess the quality of clustering, namely the Normalized Mutual Information [36] and the Adjusted Rand Index [45]. Intuitively, NMI quantifies how much the estimated clustering is informative about the true clustering, while the ARI measures the degree of agreement between an estimated clustering and a reference clustering. Higher NMI/ARI is better.

We report in Figure 3 the ARI and NMI values for the three real datasets (MNIST, PenDigits and USPS). The ARI and NMI values are given for the baseline evaluations (DAE-kmeans++ and DAE-LSC; average results over 10 runs), and the various ensemble versions of SC-EDAE (Ens.Init, Ens.Ep. and Ens.Struct.; average results over 10 runs for each of the 5 different encodings). The ensemble paradigm of SC-EDAE ensures high ARI and NMI results with low standard deviations for all real datasets, even for USPS which is an imbalanced-class dataset (Fig. 3, green boxplots).

We also detail the ARI and NMI evaluations per DAE structure in annexes, Tables 7 & 8. These supplementary results highlight the strong influence of a particular DAE structure on the ARI and NMI values. As an example, the ARI minimal and maximal values for DAE-LSC are $73.66$ and $77.75$ respectively for USPS, a difference of 4.09 (Table 7). Another striking example can be found for the SC-EDAE in its ensemble initialization version (Ens.Init.) applied to MNIST, where the ARI values fluctuate within a $[81.87;90.17]$ (Table 8). Based on these evaluations, and as already mentioned (Section 5.2), we would recommend to use SC-EDAE in its ensemble structure version (i.e., Ens.Struct.) to alleviate the issue of the DAE structure choice.

5.7 Comparison to deep k-means variants

Several strategies that use deep learning algorithm and k-means approaches, sequentially or jointly, have demonstrated accuracy improvement on the clustering task. Among these methods, two approaches can now be considered as state-of-the-art methods, namely IDEC (Improved Deep Embedded Clustering) [28] and DCN (Deep Clustering Network) [18]. Very recently, the DKM (Deep k-means) algorithm, which applies a k-means in an AE embedding space, outperformed these approaches [46].

We compare SC-EDAE to these three methods and summaries these evaluations in Table 6. The last six rows of Table 6 are directly extracted from the DKM authors study [46]. The accuracy and NMI values of these six rows are an average over 10 runs. The other values correspond to our evaluations. Specifically, baseline results are given in the first four rows, and correspond to the clustering task via k-means++ or LSC (average results over 10 runs), and via a combination of DAE and k-means or LSC (average results over 10 runs for each of the 5 different encodings). The SC-EDAE rows gives the accuracy and NMI results for our ensemble method, with an ensemble over several initializations (SC-EDAE Ens.Init.), epoch numbers (SC-EDAE Ens.Ep.) and DAE architectures (SC-EDAE Ens.Struct.).

As can be seen from Table 6, while our SC-EDAE approach does not require any pretraining, it outperforms the DCN and IDEC methods in there pretrained version (Table 6, DCNp and IDECp results). The DKM method performs well with and without pretraining. Yet, our SC-EDAE approach reaches higher accuracy and NMI results than the DKM approach with and without pretraining.

5.8 Visualization of latent space

We investigate the quality of the representation learned with SC-EDAE and in particular the positive influence of the left singular vectors matrix of $\bar{\mathbf{Z}}$, $\mathbf{B}$ (Alg.1, step c), on the clustering task. Specifically, we visualize the datapoints nearest-neighbor from the $\mathbf{B}$ matrix using the t-SNE visualization tool [47] that can project embeddings into two components (TSNE Python version from the sklearn package ). The results are given in Figure 4. The t-SNE hyperparameters perplexity, learning rate and number of iterations are set to 40, 200 and 500 for MNIST, and 25, 100 and 400 for PenDigits and USPS, following the recommendations and experimental setup of Maaten et al. [47].

For each dataset, we can observe clearly separated clusters. The ground truth labels nicely match the t-SNE datapoints gathering, highlighting the ability of SC-EDAE to separate data according to the underlying classes. As already noticed in [47], the t-SNE results obtained from the SC-EDAE ensemble affinity matrix reflects the local structure of the data, such as the orientation of the ones, by showing elongated clusters (e.g., Fig. 4, red cluster).

6 Conclusion

We report in this paper a novel clustering method that combines the advantages of deep learning, spectral clustering and ensemble strategy. Several studies have proposed to associate, either sequentially or jointly, deep architecture and classical clustering methods to improve the partitioning of large datasets. However, these methods are usually confronted to important issues related to well known challenges with neural networks, such as weight initialization or structure settings. Our SC-EDAE approach alleviates these issues by exploiting an ensemble procedure to combine several deep models before applying a spectral clustering; it is quite simple and can be framed in three steps:

generate $m$ deep embeddings from the original data,

• 2.

construct a sparse and low-dimensional ensemble affinity matrix based on anchors strategy,

• 3.

apply spectral clustering on the common space shared by the $m$ encoding.

The experiments on real and synthetic datasets demonstrate the robustness and high performance of SC-EDAE on image datasets. SC-EDAE can be used in different versions with an ensemble on weights initialization, epoch numbers or deep architectures. These variants provide higher accuracy, ARI and NMI results than state-of-the art methods. Most importantly, the high performance of SC-EDAE is obtained without any deep models pretraining.

The proposed method also benefits from the anchors strategy. The anchors provide a sparse and low-dimensional ensemble affinity matrix that ensures an efficient spectral clustering. As a complementary improvement, one could easily implements the parallelization of the $m$ encodings computation in the first step of the SC-EDAE procedure. Our experiments show that few different encodings already lead to significant performance improvement, yet more complex datasets could require larger amount of various encodings, and such parallelization would facilitate the SC-EDAE use.

Appendix A Appendix

A.1 Supplementary experiments on synthetic data

As proposed in [18], we provide two complementary examples of clustering with SC-EDAE that demonstrate the ability of the $\mathbf{B}$ embeddings to correctly recover the underlying classes of a given dataset. We first consider the following two transformations, $\mathbf{x}_{i}=\sigma(\sigma(\mathbf{W}\mathbf{h}_{i}))^{2}$ and $\mathbf{x}_{i}=\tan(\sigma(\mathbf{W}\mathbf{h}_{i}))$. The Figure 5 shows the two first embeddings of $\mathbf{B}$ obtained with the transformed data. This representation highlights the separability power of SC-EDAE. The corresponding accuracy is $1.00$ for Tetra, Chainlink and Lsun. For both supplementary transformation, we can observe patterns that are similar to clusters presented in the main text (Fig. 2).

A.2 Complementary experiments on real data

A.2.1 Baseline evaluations

The Table 7 provides complementary results for the baseline evaluations on real datasets. Specifically, it gives the mean Adjusted Rand Index (ARI) and the Normalized Mutual Information (NMI) for LSC and kmeans*++*. The mean is taken over $10$ replicates on the original datasets, over all epoch and landmark numbers. The results for DAE-LSC and DAE-kmeans++ are averaged over 50 replicates (10 replicates on each of the 5 encodings per DAE structure type), over all epoch and landmark numbers. These results follow the same trend as the accuracy results detailed in main text.

A.2.2 SC-EDAE ensemble evaluations

The Table 8 provides complementary results for the ensemble evaluations on real datasets. Specifically, it gives the mean Adjusted Rand Index (ARI) and the Normalized Mutual Information (NMI) for SC-EDAE. The mean is taken over $10$ replicates on the encodings. The columns Ens.Init. and Ens.Ep. indicate the results for an ensemble approach on the DAE weight initializations (Ens.Init., $m=5$) and the DAE training epoch numbers (Ens.Ep., $m=5$). The column Ens.Struct. provides the evaluations for an ensemble approach on various DAE structure types ($m=6$).