Generative Autoencoding of Dropout Patterns

TL;DR

This paper introduces Deciphering Autoencoders, a generative model that uses unique dropout patterns per data point to enable stable training and high-quality sampling comparable to DCGAN.

Contribution

It presents a novel autoencoder-based generative approach leveraging dropout patterns as encoding information, offering a simpler and more stable alternative to traditional generative models.

Findings

Sampling quality comparable to DCGAN on CIFAR-10

Stable training due to reliance on reconstruction error

Effective low-dimensional latent space mapping

Abstract

We propose a generative model termed Deciphering Autoencoders. In this model, we assign a unique random dropout pattern to each data point in the training dataset and then train an autoencoder to reconstruct the corresponding data point using this pattern as information to be encoded. Even if a completely random dropout pattern is assigned to each data point regardless of their similarities, a sufficiently large encoder can smoothly map them to a low-dimensional latent space to reconstruct individual training data points. During inference, using a dropout pattern different from those used during training allows the model to function as a generator. Since the training of Deciphering Autoencoders relies solely on reconstruction error, it offers more stable training compared to other generative models. Despite their simplicity, Deciphering Autoencoders show sampling quality comparable to DCGAN on the CIFAR-10 dataset.