Neural network relief: a pruning algorithm based on neural activity

TL;DR

This paper introduces a simple iterative pruning method based on neural activity importance scores, effectively reducing network size while maintaining accuracy across various architectures and datasets.

Contribution

It presents a novel importance-score based pruning algorithm that achieves high compression rates with minimal performance loss in deep neural networks.

Findings

Comparable accuracy with significantly fewer connections on LeNet/MNIST

Higher parameter compression on VGG/ResNet/CIFAR-10/100 and Tiny-ImageNet

Effective across different optimizers like Adam and SGD

Abstract

Current deep neural networks (DNNs) are overparameterized and use most of their neuronal connections during inference for each task. The human brain, however, developed specialized regions for different tasks and performs inference with a small fraction of its neuronal connections. We propose an iterative pruning strategy introducing a simple importance-score metric that deactivates unimportant connections, tackling overparameterization in DNNs and modulating the firing patterns. The aim is to find the smallest number of connections that is still capable of solving a given task with comparable accuracy, i.e. a simpler subnetwork. We achieve comparable performance for LeNet architectures on MNIST, and significantly higher parameter compression than state-of-the-art algorithms for VGG and ResNet architectures on CIFAR-10/100 and Tiny-ImageNet. Our approach also performs well for the two different optimizers considered -- Adam and SGD. The algorithm is not designed to minimize FLOPs when considering current hardware and software implementations, although it performs reasonably when compared to the state of the art.