Dropout Neural Network Training Viewed from a Percolation Perspective

TL;DR

This paper explores the percolation phenomenon in dropout training of neural networks, revealing how random connection removal affects network connectivity and can cause training breakdowns, especially in bias-free models.

Contribution

It introduces percolation models for dropout, characterizes the relationship between network topology and connectivity, and demonstrates the percolative effect's impact on training stability.

Findings

Percolation effects exist in dropout training.

Dropout can cause network disconnection and training breakdown.

Breakdowns are more severe in bias-free neural networks.

Abstract

In this work, we investigate the existence and effect of percolation in training deep Neural Networks (NNs) with dropout. Dropout methods are regularisation techniques for training NNs, first introduced by G. Hinton et al. (2012). These methods temporarily remove connections in the NN, randomly at each stage of training, and update the remaining subnetwork with Stochastic Gradient Descent (SGD). The process of removing connections from a network at random is similar to percolation, a paradigm model of statistical physics. If dropout were to remove enough connections such that there is no path between the input and output of the NN, then the NN could not make predictions informed by the data. We study new percolation models that mimic dropout in NNs and characterise the relationship between network topology and this path problem. The theory shows the existence of a percolative effect in dropout. We also show that this percolative effect can cause a breakdown when training NNs without biases with dropout; and we argue heuristically that this breakdown extends to NNs with biases.