Statistical Mechanical Analysis of Neural Network Pruning

TL;DR

This paper provides a theoretical analysis of neural network pruning techniques using statistical mechanics, demonstrating the superiority of DPP node pruning and the better generalization of sparse networks over dense ones.

Contribution

It introduces a theoretical framework for understanding pruning methods and proves the advantages of DPP node pruning and sparse networks over dense ones.

Findings

DPP node pruning outperforms other methods on real datasets.

Sparse neural networks generalize better than dense networks for the same parameter count.

Random edge pruning can outperform DPP node pruning in certain scenarios.

Abstract

Deep learning architectures with a huge number of parameters are often compressed using pruning techniques to ensure computational efficiency of inference during deployment. Despite multitude of empirical advances, there is a lack of theoretical understanding of the effectiveness of different pruning methods. We inspect different pruning techniques under the statistical mechanics formulation of a teacher-student framework and derive their generalization error (GE) bounds. It has been shown that Determinantal Point Process (DPP) based node pruning method is notably superior to competing approaches when tested on real datasets. Using GE bounds in the aforementioned setup we provide theoretical guarantees for their empirical observations. Another consistent finding in literature is that sparse neural networks (edge pruned) generalize better than dense neural networks (node pruned) for a fixed number of parameters. We use our theoretical setup to prove this finding and show that even the baseline random edge pruning method performs better than the DPP node pruning method. We also validate this empirically on real datasets.