Gradual Training Method for Denoising Auto Encoders

TL;DR

This paper introduces a gradual training scheme for deep denoising autoencoders that incrementally adds layers, leading to improved reconstruction and classification performance on mid-sized datasets like MNIST and CIFAR.

Contribution

The paper proposes a novel gradual training method for deep DAEs, enhancing their effectiveness over traditional stacked training methods.

Findings

Gradual training improves reconstruction quality.

Gradual training reduces classification error.

Method shows consistent gains on MNIST and CIFAR datasets.

Abstract

Stacked denoising auto encoders (DAEs) are well known to learn useful deep representations, which can be used to improve supervised training by initializing a deep network. We investigate a training scheme of a deep DAE, where DAE layers are gradually added and keep adapting as additional layers are added. We show that in the regime of mid-sized datasets, this gradual training provides a small but consistent improvement over stacked training in both reconstruction quality and classification error over stacked training on MNIST and CIFAR datasets.