Gradual training of deep denoising auto encoders

TL;DR

This paper introduces a gradual training scheme for deep denoising autoencoders that incrementally adds layers and adapts them, leading to improved reconstruction and classification performance on mid-sized datasets like MNIST and CIFAR.

Contribution

It proposes a novel incremental training method for deep DAEs that enhances performance over traditional stacked training methods.

Findings

Gradual training improves reconstruction quality.

Gradual training reduces classification error.

Performance gains are consistent on MNIST and CIFAR datasets.

Abstract

Stacked denoising auto encoders (DAEs) are well known to learn useful deep representations, which can be used to improve supervised training by initializing a deep network. We investigate a training scheme of a deep DAE, where DAE layers are gradually added and keep adapting as additional layers are added. We show that in the regime of mid-sized datasets, this gradual training provides a small but consistent improvement over stacked training in both reconstruction quality and classification error over stacked training on MNIST and CIFAR datasets.