On denoising autoencoders trained to minimise binary cross-entropy

TL;DR

This paper demonstrates that denoising autoencoders trained with binary cross-entropy can be theoretically used to move data samples towards higher probability regions, enabling data synthesis and sample improvement.

Contribution

It provides a theoretical foundation for using BCE-trained DAEs for data distribution exploration, extending previous results from MSE-based DAEs.

Findings

DAEs trained with BCE can perform gradient steps towards high-probability data regions.

Iterative application of BCE-trained DAEs can synthesize new data samples from noise.

DAEs can improve initial data samples generated by other methods.

Abstract

Denoising autoencoders (DAEs) are powerful deep learning models used for feature extraction, data generation and network pre-training. DAEs consist of an encoder and decoder which may be trained simultaneously to minimise a loss (function) between an input and the reconstruction of a corrupted version of the input. There are two common loss functions used for training autoencoders, these include the mean-squared error (MSE) and the binary cross-entropy (BCE). When training autoencoders on image data a natural choice of loss function is BCE, since pixel values may be normalised to take values in [0,1] and the decoder model may be designed to generate samples that take values in (0,1). We show theoretically that DAEs trained to minimise BCE may be used to take gradient steps in the data space towards regions of high probability under the data-generating distribution. Previously this had only been shown for DAEs trained using MSE. As a consequence of the theory, iterative application of a trained DAE moves a data sample from regions of low probability to regions of higher probability under the data-generating distribution. Firstly, we validate the theory by showing that novel data samples, consistent with the training data, may be synthesised when the initial data samples are random noise. Secondly, we motivate the theory by showing that initial data samples synthesised via other methods may be improved via iterative application of a trained DAE to those initial samples.