Criticality versus uniformity in deep neural networks

TL;DR

This paper investigates how saturation of the tanh activation function along the edge of chaos affects the trainability of deep neural networks, revealing that optimal initialization requires more than just being on the edge of chaos.

Contribution

It identifies the line of uniformity in phase space where activation entropy is maximized and shows its intersection with the edge of chaos, highlighting the limits of current initialization strategies.

Findings

Saturation impedes training efficiency beyond the line of uniformity.

Initialization on the edge of chaos is necessary but not sufficient for optimal training.

Maximum entropy in activation distribution occurs along the line of uniformity.

Abstract

Deep feedforward networks initialized along the edge of chaos exhibit exponentially superior training ability as quantified by maximum trainable depth. In this work, we explore the effect of saturation of the tanh activation function along the edge of chaos. In particular, we determine the line of uniformity in phase space along which the post-activation distribution has maximum entropy. This line intersects the edge of chaos, and indicates the regime beyond which saturation of the activation function begins to impede training efficiency. Our results suggest that initialization along the edge of chaos is a necessary but not sufficient condition for optimal trainability.