Information Geometry of Dropout Training

TL;DR

This paper analyzes dropout regularization in neural networks through the lens of information geometry, revealing its effects on model manifold curvature and Fisher information-based regularization, supported by theoretical and numerical insights.

Contribution

It provides a unified geometric perspective on dropout, connecting its regularization effects to model manifold curvature and Fisher information, offering new theoretical understanding.

Findings

Dropout flattens the model manifold.

Regularization strength depends on curvature.

Dropout relates to Fisher information-based regularization.

Abstract

Dropout is one of the most popular regularization techniques in neural network training. Because of its power and simplicity of idea, dropout has been analyzed extensively and many variants have been proposed. In this paper, several properties of dropout are discussed in a unified manner from the viewpoint of information geometry. We showed that dropout flattens the model manifold and that their regularization performance depends on the amount of the curvature. Then, we showed that dropout essentially corresponds to a regularization that depends on the Fisher information, and support this result from numerical experiments. Such a theoretical analysis of the technique from a different perspective is expected to greatly assist in the understanding of neural networks, which are still in their infancy.