Mean Field Theory of Activation Functions in Deep Neural Networks

TL;DR

This paper develops a statistical mechanics model of deep neural networks to understand activation functions, revealing how different activations influence information propagation and network performance.

Contribution

It introduces a mean field theory linking energy-based and feedforward networks, deriving natural activation functions and analyzing their impact on network behavior.

Findings

ReLU emerges as a zero-noise limit of the model

Swish activation performs more consistently across architectures

Spectrum analysis shows activation functions affect Hessian properties

Abstract

We present a Statistical Mechanics (SM) model of deep neural networks, connecting the energy-based and the feed forward networks (FFN) approach. We infer that FFN can be understood as performing three basic steps: encoding, representation validation and propagation. From the meanfield solution of the model, we obtain a set of natural activations -- such as Sigmoid, $\tanh$ and ReLu -- together with the state-of-the-art, Swish; this represents the expected information propagating through the network and tends to ReLu in the limit of zero noise.We study the spectrum of the Hessian on an associated classification task, showing that Swish allows for more consistent performances over a wider range of network architectures.