Characterizing Sparse Connectivity Patterns in Neural Networks

TL;DR

This paper introduces a method for reducing neural network parameters by using predefined sparsity, achieving comparable accuracy with significantly fewer connections, and introduces a new metric to evaluate connection patterns.

Contribution

It presents a novel sparsity technique for neural networks, demonstrating minimal accuracy loss at very low connection densities and introducing the 'scatter' metric for pattern quality assessment.

Findings

Networks can operate with less than 0.5% connection density without accuracy loss.

Predefined sparsity can be effectively applied to fully connected layers.

The 'scatter' metric helps characterize and evaluate connection pattern quality.

Abstract

We propose a novel way of reducing the number of parameters in the storage-hungry fully connected layers of a neural network by using pre-defined sparsity, where the majority of connections are absent prior to starting training. Our results indicate that convolutional neural networks can operate without any loss of accuracy at less than half percent classification layer connection density, or less than 5 percent overall network connection density. We also investigate the effects of pre-defining the sparsity of networks with only fully connected layers. Based on our sparsifying technique, we introduce the `scatter' metric to characterize the quality of a particular connection pattern. As proof of concept, we show results on CIFAR, MNIST and a new dataset on classifying Morse code symbols, which highlights some interesting trends and limits of sparse connection patterns.