Phase Transitions in Neural Networks Pruning

TL;DR

This paper analyzes neural network pruning through a statistical physics lens, revealing a phase transition from functional to disordered states, characterized by critical scaling laws and topological properties.

Contribution

It introduces a phase transition framework for understanding pruning effects, highlighting universal critical behavior and topological factors influencing network performance.

Findings

Pruning induces a sharp phase transition in network performance.

Scaling laws near the transition point resemble second-order critical phenomena.

Subnetwork topologies significantly impact learning ability and robustness.

Abstract

Deep neural networks are strongly over-parameterized, often containing far more weights than required for their task. Although such redundancy can aid optimization, it leads to inefficient deployment and high computational cost, motivating model compression techniques. Among these, network pruning provides a clear and effective route to sparsity. We study pruning from a statistical-physics perspective, interpreting performance degradation under weight removal as a phase transition. Focusing on magnitude-based pruning with fine-tuning, we show that deep networks undergo a sharp transition from a cooperative, functional phase to a disordered phase with collapsed performance. This transition is characterized by scaling laws consistent with second-order critical behavior, with connectivity as the control parameter. Our findings suggest universal pruning-induced criticality across architectures and datasets. Finally, we show that there exists a large class of subnetworks sharing the same nodes' degrees with similar learning ability, thus linking model performance to its topological properties.