Scaling and Resizing Symmetry in Feedforward Networks

TL;DR

This paper explores how criticality and scaling symmetries in weight initialization influence training efficiency in deep neural networks, revealing new invariances related to data resizing.

Contribution

It demonstrates the presence of physical system-like scaling symmetry at criticality in untrained networks and introduces a new data-resizing symmetry derived from this.

Findings

Scaling symmetry at criticality affects network behavior.

Data-resizing symmetry is inherited from critical scaling.

Improved understanding of initialization impacts training speed.

Abstract

Weights initialization in deep neural networks have a strong impact on the speed of converge of the learning map. Recent studies have shown that in the case of random initializations, a chaos/order phase transition occur in the space of variances of random weights and biases. Experiments then had shown that large improvements can be made, in terms of the training speed, if a neural network is initialized on values along the critical line of such phase transition. In this contribution, we show evidence that the scaling property exhibited by physical systems at criticality, is also present in untrained feedforward networks with random weights initialization at the critical line. Additionally, we suggest an additional data-resizing symmetry, which is directly inherited from the scaling symmetry at criticality.