Stochastic Modified Equations and Dynamics of Dropout Algorithm

TL;DR

This paper derives stochastic modified equations to analyze dropout's dynamics in neural network training, revealing how dropout promotes convergence to flatter minima through noise structure analysis.

Contribution

It introduces a novel stochastic differential equation framework for dropout, providing new theoretical insights into its role in finding flatter minima.

Findings

Dropout's noise structure relates inverse variance to flatness of minima.

Empirical evidence supports the inverse variance-flatness relation.

Dropout tends to locate flatter minima during training.

Abstract

Dropout is a widely utilized regularization technique in the training of neural networks, nevertheless, its underlying mechanism and its impact on achieving good generalization abilities remain poorly understood. In this work, we derive the stochastic modified equations for analyzing the dynamics of dropout, where its discrete iteration process is approximated by a class of stochastic differential equations. In order to investigate the underlying mechanism by which dropout facilitates the identification of flatter minima, we study the noise structure of the derived stochastic modified equation for dropout. By drawing upon the structural resemblance between the Hessian and covariance through several intuitive approximations, we empirically demonstrate the universal presence of the inverse variance-flatness relation and the Hessian-variance relation, throughout the training process of dropout. These theoretical and empirical findings make a substantial contribution to our understanding of the inherent tendency of dropout to locate flatter minima.