The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective

TL;DR

This paper investigates how increasing width in neural networks, through the lens of Deep Gaussian Processes, can actually hinder performance by causing models to behave more like Gaussian processes, revealing an optimal width for best results.

Contribution

It provides a theoretical and empirical analysis showing that large width can be detrimental, identifying a 'sweet spot' for width in Deep Gaussian Processes and relating findings to conventional neural networks.

Findings

Large width causes Deep GP models to converge to Gaussian processes.

Optimal width for maximum performance is around 1 or 2 units.

Further increasing width beyond the optimal degrades performance.

Abstract

Large width limits have been a recent focus of deep learning research: modulo computational practicalities, do wider networks outperform narrower ones? Answering this question has been challenging, as conventional networks gain representational power with width, potentially masking any negative effects. Our analysis in this paper decouples capacity and width via the generalization of neural networks to Deep Gaussian Processes (Deep GP), a class of nonparametric hierarchical models that subsume neural nets. In doing so, we aim to understand how width affects (standard) neural networks once they have sufficient capacity for a given modeling task. Our theoretical and empirical results on Deep GP suggest that large width can be detrimental to hierarchical models. Surprisingly, we prove that even nonparametric Deep GP converge to Gaussian processes, effectively becoming shallower without any increase in representational power. The posterior, which corresponds to a mixture of data-adaptable basis functions, becomes less data-dependent with width. Our tail analysis demonstrates that width and depth have opposite effects: depth accentuates a model's non-Gaussianity, while width makes models increasingly Gaussian. We find there is a "sweet spot" that maximizes test performance before the limiting GP behavior prevents adaptability, occurring at width = 1 or width = 2 for nonparametric Deep GP. These results make strong predictions about the same phenomenon in conventional neural networks trained with L2 regularization (analogous to a Gaussian prior on parameters): we show that such neural networks may need up to 500 - 1000 hidden units for sufficient capacity - depending on the dataset - but further width degrades performance.