Towards Dropout Training for Convolutional Neural Networks

TL;DR

This paper investigates dropout in convolutional neural networks, proposing probabilistic weighted pooling as an alternative to max-pooling, leading to improved performance and insights into dropout effects in convolutional and pooling layers.

Contribution

It introduces probabilistic weighted pooling as a test-time alternative to max-pooling and demonstrates its effectiveness, along with a comprehensive analysis of dropout in convolutional layers.

Findings

Probabilistic weighted pooling outperforms max-pooling.

Dropout in convolutional layers has a significant impact.

Achieved state-of-the-art on MNIST, competitive on CIFAR datasets.

Abstract

Recently, dropout has seen increasing use in deep learning. For deep convolutional neural networks, dropout is known to work well in fully-connected layers. However, its effect in convolutional and pooling layers is still not clear. This paper demonstrates that max-pooling dropout is equivalent to randomly picking activation based on a multinomial distribution at training time. In light of this insight, we advocate employing our proposed probabilistic weighted pooling, instead of commonly used max-pooling, to act as model averaging at test time. Empirical evidence validates the superiority of probabilistic weighted pooling. We also empirically show that the effect of convolutional dropout is not trivial, despite the dramatically reduced possibility of over-fitting due to the convolutional architecture. Elaborately designing dropout training simultaneously in max-pooling and fully-connected layers, we achieve state-of-the-art performance on MNIST, and very competitive results on CIFAR-10 and CIFAR-100, relative to other approaches without data augmentation. Finally, we compare max-pooling dropout and stochastic pooling, both of which introduce stochasticity based on multinomial distributions at pooling stage.