Frivolous Units: Wider Networks Are Not Really That Wide

TL;DR

This paper investigates the role of 'frivolous' units in overparameterized neural networks, revealing how their proliferation explains why increasing network width does not impair accuracy and shedding light on implicit regularization.

Contribution

It identifies two types of frivolous units—prunable and redundant—that emerge with increased width and explains their influence on network complexity and regularization.

Findings

Frivolous units proliferate as network width increases.

Prunable units can be removed with minimal impact on output.

Redundant units' activities are linear combinations of others.

Abstract

A remarkable characteristic of overparameterized deep neural networks (DNNs) is that their accuracy does not degrade when the network's width is increased. Recent evidence suggests that developing compressible representations is key for adjusting the complexity of large networks to the learning task at hand. However, these compressible representations are poorly understood. A promising strand of research inspired from biology is understanding representations at the unit level as it offers a more granular and intuitive interpretation of the neural mechanisms. In order to better understand what facilitates increases in width without decreases in accuracy, we ask: Are there mechanisms at the unit level by which networks control their effective complexity as their width is increased? If so, how do these depend on the architecture, dataset, and training parameters? We identify two distinct types of "frivolous" units that proliferate when the network's width is increased: prunable units which can be dropped out of the network without significant change to the output and redundant units whose activities can be expressed as a linear combination of others. These units imply complexity constraints as the function the network represents could be expressed by a network without them. We also identify how the development of these units can be influenced by architecture and a number of training factors. Together, these results help to explain why the accuracy of DNNs does not degrade when width is increased and highlight the importance of frivolous units toward understanding implicit regularization in DNNs.