Rank Reduction Autoencoders

TL;DR

This paper introduces Rank Reduction Autoencoders (RRAEs), a new deterministic autoencoder class that uses SVD-based regularization of the latent space, enabling adaptive bottleneck size selection and improved performance on various datasets.

Contribution

The paper proposes RRAEs, which regularize latent spaces via SVD, and introduces aRRAEs, an adaptive algorithm for optimal bottleneck size determination during training.

Findings

RRAEs outperform vanilla autoencoders on multiple datasets.

RRAEs are stable, scalable, and hyperparameter-free.

Adaptive aRRAEs effectively determine optimal bottleneck sizes.

Abstract

The choice of an appropriate bottleneck dimension and the application of effective regularization are both essential for Autoencoders to learn meaningful representations from unlabeled data. In this paper, we introduce a new class of deterministic autoencoders, Rank Reduction Autoencoders (RRAEs), which regularize their latent spaces by employing a truncated singular value decomposition (SVD) during training. In RRAEs, the bottleneck is defined by the rank of the latent matrix, thereby alleviating the dependence of the encoder/decoder architecture on the bottleneck size. This approach enabled us to propose an adaptive algorithm (aRRAEs) that efficiently determines the optimal bottleneck size during training. We empirically demonstrate that both RRAEs and aRRAEs are stable, scalable, and reliable, as they do not introduce any additional training hyperparameters. We evaluate our proposed architecture on a synthetic data set, as well as on MNIST, Fashion MNIST, and CelebA. Our results show that RRAEs offer several advantages over Vanilla AEs with both large and small latent spaces, and outperform other regularizing AE architectures.