A Probabilistic Approach to Neural Network Pruning

TL;DR

This paper provides a probabilistic theoretical analysis of neural network pruning techniques, demonstrating that pruned networks can retain expressive power close to the original, with bounds on performance gaps.

Contribution

It introduces a universal probabilistic framework to analyze the capabilities of random and magnitude-based pruning methods for FCNs and CNNs.

Findings

Pruned networks can approximate the target network within specified bounds.

Theoretical bounds are established for the expressive power of pruned networks.

Analysis applies to networks with weights sampled from appropriate distributions.

Abstract

Neural network pruning techniques reduce the number of parameters without compromising predicting ability of a network. Many algorithms have been developed for pruning both over-parameterized fully-connected networks (FCNs) and convolutional neural networks (CNNs), but analytical studies of capabilities and compression ratios of such pruned sub-networks are lacking. We theoretically study the performance of two pruning techniques (random and magnitude-based) on FCNs and CNNs. Given a target network {whose weights are independently sampled from appropriate distributions}, we provide a universal approach to bound the gap between a pruned and the target network in a probabilistic sense. The results establish that there exist pruned networks with expressive power within any specified bound from the target network.