Activation function dependence of the storage capacity of treelike neural networks

TL;DR

This paper investigates how different activation functions influence the storage capacity of treelike neural networks, revealing that nonlinearity impacts both capacity and robustness, and offering estimates for common functions.

Contribution

It provides a detailed analysis of the relationship between activation function smoothness and network capacity, connecting theoretical insights to biological neuron behavior.

Findings

Nonlinearity can increase storage capacity.

Nonlinearity can decrease robustness.

Provides estimates for common activation functions.

Abstract

The expressive power of artificial neural networks crucially depends on the nonlinearity of their activation functions. Though a wide variety of nonlinear activation functions have been proposed for use in artificial neural networks, a detailed understanding of their role in determining the expressive power of a network has not emerged. Here, we study how activation functions affect the storage capacity of treelike two-layer networks. We relate the boundedness or divergence of the capacity in the infinite-width limit to the smoothness of the activation function, elucidating the relationship between previously studied special cases. Our results show that nonlinearity can both increase capacity and decrease the robustness of classification, and provide simple estimates for the capacity of networks with several commonly used activation functions. Furthermore, they generate a hypothesis for the functional benefit of dendritic spikes in branched neurons.