Are We Using Autoencoders in a Wrong Way?

TL;DR

This paper critically examines the standard training of autoencoders by modifying the latent space shape and reconstructing different class samples, revealing new insights into their behavior and potential misuse.

Contribution

The study introduces a novel approach to training autoencoders by altering the latent space and reconstructing different class samples without explicit regularization.

Findings

Latent space shape significantly affects autoencoder performance.

Reconstructing different class samples offers new perspectives on latent representations.

Modifying latent space can lead to better understanding of autoencoder behavior.

Abstract

Autoencoders are certainly among the most studied and used Deep Learning models: the idea behind them is to train a model in order to reconstruct the same input data. The peculiarity of these models is to compress the information through a bottleneck, creating what is called Latent Space. Autoencoders are generally used for dimensionality reduction, anomaly detection and feature extraction. These models have been extensively studied and updated, given their high simplicity and power. Examples are (i) the Denoising Autoencoder, where the model is trained to reconstruct an image from a noisy one; (ii) Sparse Autoencoder, where the bottleneck is created by a regularization term in the loss function; (iii) Variational Autoencoder, where the latent space is used to generate new consistent data. In this article, we revisited the standard training for the undercomplete Autoencoder modifying the shape of the latent space without using any explicit regularization term in the loss function. We forced the model to reconstruct not the same observation in input, but another one sampled from the same class distribution. We also explored the behaviour of the latent space in the case of reconstruction of a random sample from the whole dataset.