Stochastic Neural Networks with Infinite Width are Deterministic

TL;DR

This paper proves that as the width of an optimized stochastic neural network approaches infinity, its predictive variance diminishes, providing theoretical insight into how stochasticity regularizes neural networks and connecting to dropout and Bayesian models.

Contribution

It offers a theoretical analysis showing that infinite-width stochastic neural networks become deterministic, clarifying the role of stochasticity in regularization.

Findings

Predictive variance tends to zero as width increases

Stochasticity acts as a regularizer through averaging effects

Results apply to dropout and Bayesian latent variable models

Abstract

This work theoretically studies stochastic neural networks, a main type of neural network in use. We prove that as the width of an optimized stochastic neural network tends to infinity, its predictive variance on the training set decreases to zero. Our theory justifies the common intuition that adding stochasticity to the model can help regularize the model by introducing an averaging effect. Two common examples that our theory can be relevant to are neural networks with dropout and Bayesian latent variable models in a special limit. Our result thus helps better understand how stochasticity affects the learning of neural networks and potentially design better architectures for practical problems.