On the Implicit Bias of Dropout

TL;DR

This paper investigates the implicit bias introduced by dropout in deep learning, revealing how it equalizes weight norms in single-layer linear networks and characterizing the associated optimization landscape.

Contribution

It provides a theoretical analysis of dropout's implicit bias, specifically showing norm equalization and detailing the optimization landscape for single-layer linear neural networks.

Findings

Dropout induces equal norms in incoming and outgoing weights of hidden nodes.

Complete characterization of the dropout-induced optimization landscape.

Insights into how dropout contributes to generalization in over-parametrized models.

Abstract

Algorithmic approaches endow deep learning systems with implicit bias that helps them generalize even in over-parametrized settings. In this paper, we focus on understanding such a bias induced in learning through dropout, a popular technique to avoid overfitting in deep learning. For single hidden-layer linear neural networks, we show that dropout tends to make the norm of incoming/outgoing weight vectors of all the hidden nodes equal. In addition, we provide a complete characterization of the optimization landscape induced by dropout.