Survey of Dropout Methods for Deep Neural Networks

TL;DR

This survey reviews the development, applications, and recent advances of dropout methods in deep neural networks, highlighting their role in regularization, model compression, and uncertainty estimation across various network architectures.

Contribution

It provides a comprehensive overview of dropout techniques, their evolution, and current research directions, including detailed descriptions of key methods.

Findings

Dropout is effective for neural network regularization.

Recent advances extend dropout to convolutional and recurrent layers.

Dropout techniques are widely used in practice for various applications.

Abstract

Dropout methods are a family of stochastic techniques used in neural network training or inference that have generated significant research interest and are widely used in practice. They have been successfully applied in neural network regularization, model compression, and in measuring the uncertainty of neural network outputs. While original formulated for dense neural network layers, recent advances have made dropout methods also applicable to convolutional and recurrent neural network layers. This paper summarizes the history of dropout methods, their various applications, and current areas of research interest. Important proposed methods are described in additional detail.