A Signal Propagation Perspective for Pruning Neural Networks at Initialization

TL;DR

This paper provides a theoretical analysis of why pruning neural networks at initialization works, introduces a signal propagation perspective to improve pruning methods, and demonstrates enhanced results across various models and scenarios.

Contribution

It offers a formal characterization of initialization conditions for effective pruning, analyzes signal propagation in pruned networks, and proposes a data-free method to enhance trainability.

Findings

Improved pruning results on multiple image classification models.

A formal link between connection sensitivity and gradient-based measures.

Enhanced trainability of pruned networks through signal propagation analysis.

Abstract

Network pruning is a promising avenue for compressing deep neural networks. A typical approach to pruning starts by training a model and then removing redundant parameters while minimizing the impact on what is learned. Alternatively, a recent approach shows that pruning can be done at initialization prior to training, based on a saliency criterion called connection sensitivity. However, it remains unclear exactly why pruning an untrained, randomly initialized neural network is effective. In this work, by noting connection sensitivity as a form of gradient, we formally characterize initialization conditions to ensure reliable connection sensitivity measurements, which in turn yields effective pruning results. Moreover, we analyze the signal propagation properties of the resulting pruned networks and introduce a simple, data-free method to improve their trainability. Our modifications to the existing pruning at initialization method lead to improved results on all tested network models for image classification tasks. Furthermore, we empirically study the effect of supervision for pruning and demonstrate that our signal propagation perspective, combined with unsupervised pruning, can be useful in various scenarios where pruning is applied to non-standard arbitrarily-designed architectures.