Criticality & Deep Learning I: Generally Weighted Nets

TL;DR

This paper explores the role of criticality and phase transitions in deep learning, combining theoretical physics calculations with experimental analysis to understand their impact on neural network performance.

Contribution

It introduces a theoretical framework for analyzing critical points in deep networks and provides initial experimental evidence of criticality in neural models.

Findings

Critical points can be calculated using statistical physics methods.

Evidence of criticality traces found in deep neural networks.

Lays groundwork for future studies on criticality and learning in deep networks.

Abstract

Motivated by the idea that criticality and universality of phase transitions might play a crucial role in achieving and sustaining learning and intelligent behaviour in biological and artificial networks, we analyse a theoretical and a pragmatic experimental set up for critical phenomena in deep learning. On the theoretical side, we use results from statistical physics to carry out critical point calculations in feed-forward/fully connected networks, while on the experimental side we set out to find traces of criticality in deep neural networks. This is our first step in a series of upcoming investigations to map out the relationship between criticality and learning in deep networks.