Reducing Redundancy in the Bottleneck Representation of the Autoencoders

TL;DR

This paper introduces a novel loss term based on pair-wise neuron correlation to explicitly reduce redundancy in autoencoder bottleneck representations, improving their diversity and performance across various tasks.

Contribution

It proposes a new explicit redundancy penalty in autoencoders, enhancing the diversity of learned representations beyond standard reconstruction loss.

Findings

Improved performance in dimensionality reduction tasks.

Enhanced image compression results on MNIST.

Better denoising capabilities on Fashion MNIST.

Abstract

Autoencoders are a type of unsupervised neural networks, which can be used to solve various tasks, e.g., dimensionality reduction, image compression, and image denoising. An AE has two goals: (i) compress the original input to a low-dimensional space at the bottleneck of the network topology using an encoder, (ii) reconstruct the input from the representation at the bottleneck using a decoder. Both encoder and decoder are optimized jointly by minimizing a distortion-based loss which implicitly forces the model to keep only those variations of input data that are required to reconstruct the and to reduce redundancies. In this paper, we propose a scheme to explicitly penalize feature redundancies in the bottleneck representation. To this end, we propose an additional loss term, based on the pair-wise correlation of the neurons, which complements the standard reconstruction loss forcing the encoder to learn a more diverse and richer representation of the input. We tested our approach across different tasks: dimensionality reduction using three different dataset, image compression using the MNIST dataset, and image denoising using fashion MNIST. The experimental results show that the proposed loss leads consistently to superior performance compared to the standard AE loss.