Pruning-induced phases in fully-connected neural networks: the eumentia, the dementia, and the amentia

TL;DR

This paper investigates how pruning via dropout induces phase transitions in neural networks, identifying three phases with distinct learning behaviors and revealing a BKT-like transition, thus connecting neural network dynamics with statistical mechanics.

Contribution

The study uncovers phase transitions in neural networks caused by pruning, characterizes their universality class, and links neural scaling laws to statistical mechanics phenomena.

Findings

Identified three phases: eumentia, dementia, and amentia.

Discovered a BKT-like phase transition between learning and forgetting.

Showed the phase structure is robust across network architectures.

Abstract

Modern neural networks are heavily overparameterized, and pruning, which removes redundant neurons or connections, has emerged as a key approach to compressing them without sacrificing performance. However, while practical pruning methods are well developed, whether pruning induces sharp phase transitions in the neural networks and, if so, to what universality class they belong, remain open questions. To address this, we study fully-connected neural networks trained on MNIST, independently varying the dropout (i.e., removing neurons) rate at both the training and evaluation stages to map the phase diagram. We identify three distinct phases: eumentia (the network learns), dementia (the network has forgotten), and amentia (the network cannot learn), sharply distinguished by the power-law scaling of the cross-entropy loss with the training dataset size. {In the eumentia phase, the algebraic decay of the loss, as documented in the machine learning literature as neural scaling laws, is from the perspective of statistical mechanics the hallmark of quasi-long-range order.} We demonstrate that the transition between the eumentia and dementia phases is accompanied by scale invariance, with a diverging length scale that exhibits hallmarks of a Berezinskii-Kosterlitz-Thouless-like transition; the phase structure is robust across different network widths and depths. Our results establish that dropout-induced pruning provides a concrete setting in which neural network behavior can be understood through the lens of statistical mechanics.