Slimming Neural Networks using Adaptive Connectivity Scores

TL;DR

SNACS is a fast, automated neural network pruning method that combines probabilistic and constraint-based approaches, using adaptive connectivity scores to efficiently prune filters while preserving important ones, achieving state-of-the-art results.

Contribution

The paper introduces SNACS, a novel single-shot pruning algorithm that integrates a probabilistic framework with constraints via a new connectivity measure, improving speed and accuracy.

Findings

SNACS is over 17 times faster than comparable methods.

SNACS achieves state-of-the-art pruning performance on multiple benchmarks.

SNACS effectively identifies and preserves critical filters during pruning.

Abstract

In general, deep neural network (DNN) pruning methods fall into two categories: 1) Weight-based deterministic constraints, and 2) Probabilistic frameworks. While each approach has its merits and limitations there are a set of common practical issues such as, trial-and-error to analyze sensitivity and hyper-parameters to prune DNNs, which plague them both. In this work, we propose a new single-shot, fully automated pruning algorithm called Slimming Neural networks using Adaptive Connectivity Scores (SNACS). Our proposed approach combines a probabilistic pruning framework with constraints on the underlying weight matrices, via a novel connectivity measure, at multiple levels to capitalize on the strengths of both approaches while solving their deficiencies. In \alg{}, we propose a fast hash-based estimator of Adaptive Conditional Mutual Information (ACMI), that uses a weight-based scaling criterion, to evaluate the connectivity between filters and prune unimportant ones. To automatically determine the limit up to which a layer can be pruned, we propose a set of operating constraints that jointly define the upper pruning percentage limits across all the layers in a deep network. Finally, we define a novel sensitivity criterion for filters that measures the strength of their contributions to the succeeding layer and highlights critical filters that need to be completely protected from pruning. Through our experimental validation we show that SNACS is faster by over 17x the nearest comparable method and is the state of the art single-shot pruning method across three standard Dataset-DNN pruning benchmarks: CIFAR10-VGG16, CIFAR10-ResNet56 and ILSVRC2012-ResNet50.