Multi-Sample Dropout for Accelerated Training and Better Generalization

TL;DR

This paper introduces multi-sample dropout, an enhanced regularization method that creates multiple dropout samples per iteration, leading to faster training and improved generalization in deep neural networks.

Contribution

It proposes a simple extension to dropout that averages losses over multiple samples, accelerating training and enhancing model performance without significant additional computational cost.

Findings

Multi-sample dropout accelerates training on image classification tasks.

Networks with multi-sample dropout achieve lower error rates than with original dropout.

Additional computation cost is minimal due to shared weights and convolutional layer dominance.

Abstract

Dropout is a simple but efficient regularization technique for achieving better generalization of deep neural networks (DNNs); hence it is widely used in tasks based on DNNs. During training, dropout randomly discards a portion of the neurons to avoid overfitting. This paper presents an enhanced dropout technique, which we call multi-sample dropout, for both accelerating training and improving generalization over the original dropout. The original dropout creates a randomly selected subset (called a dropout sample) from the input in each training iteration while the multi-sample dropout creates multiple dropout samples. The loss is calculated for each sample, and then the sample losses are averaged to obtain the final loss. This technique can be easily implemented by duplicating a part of the network after the dropout layer while sharing the weights among the duplicated fully connected layers. Experimental results using image classification tasks including ImageNet, CIFAR-10, and CIFAR-100 showed that multi-sample dropout accelerates training. Moreover, the networks trained using multi-sample dropout achieved lower error rates compared to networks trained with the original dropout. The additional computation cost due to the duplicated operations is not significant for deep convolutional networks because most of the computation time is consumed in the convolution layers before the dropout layer, which are not duplicated.