Joint Optimization of an Autoencoder for Clustering and Embedding

TL;DR

This paper introduces a novel deep clustering approach that jointly optimizes an autoencoder and clustering, leveraging a theoretical link to Gaussian mixture models to improve unsupervised categorization.

Contribution

It presents a unified deep clustering model that learns embeddings and clusters simultaneously, based on a theoretical connection between GMMs and autoencoder loss functions.

Findings

Outperforms baseline methods on multiple datasets

Theoretical equivalence between GMMs and autoencoder-based clustering

Joint optimization improves clustering quality

Abstract

Deep embedded clustering has become a dominating approach to unsupervised categorization of objects with deep neural networks. The optimization of the most popular methods alternates between the training of a deep autoencoder and a k-means clustering of the autoencoder's embedding. The diachronic setting, however, prevents the former to benefit from valuable information acquired by the latter. In this paper, we present an alternative where the autoencoder and the clustering are learned simultaneously. This is achieved by providing novel theoretical insight, where we show that the objective function of a certain class of Gaussian mixture models (GMMs) can naturally be rephrased as the loss function of a one-hidden layer autoencoder thus inheriting the built-in clustering capabilities of the GMM. That simple neural network, referred to as the clustering module, can be integrated into a deep autoencoder resulting in a deep clustering model able to jointly learn a clustering and an embedding. Experiments confirm the equivalence between the clustering module and Gaussian mixture models. Further evaluations affirm the empirical relevance of our deep architecture as it outperforms related baselines on several data sets.