Weight Expansion: A New Perspective on Dropout and Generalization

TL;DR

This paper introduces the concept of weight expansion as a key mechanism behind dropout's success in improving neural network generalization, supported by theoretical and empirical evidence.

Contribution

It establishes weight expansion as a fundamental factor in dropout's effectiveness and suggests it as a target for designing better regularizers.

Findings

Dropout induces weight expansion in neural networks.

Weight expansion correlates with improved generalization.

Methods that increase weight expansion tend to enhance generalization.

Abstract

While dropout is known to be a successful regularization technique, insights into the mechanisms that lead to this success are still lacking. We introduce the concept of \emph{weight expansion}, an increase in the signed volume of a parallelotope spanned by the column or row vectors of the weight covariance matrix, and show that weight expansion is an effective means of increasing the generalization in a PAC-Bayesian setting. We provide a theoretical argument that dropout leads to weight expansion and extensive empirical support for the correlation between dropout and weight expansion. To support our hypothesis that weight expansion can be regarded as an \emph{indicator} of the enhanced generalization capability endowed by dropout, and not just as a mere by-product, we have studied other methods that achieve weight expansion (resp.\ contraction), and found that they generally lead to an increased (resp.\ decreased) generalization ability. This suggests that dropout is an attractive regularizer, because it is a computationally cheap method for obtaining weight expansion. This insight justifies the role of dropout as a regularizer, while paving the way for identifying regularizers that promise improved generalization through weight expansion.