The Power of Sparsity in Convolutional Neural Networks

TL;DR

This paper explores the use of sparsity in convolutional neural networks, demonstrating that channel-wise sparse connections can significantly reduce computational costs while maintaining accuracy, especially in large architectures.

Contribution

It introduces a generalized channel-wise sparse connection strategy for convolutional layers, improving efficiency over simple channel reduction methods.

Findings

Sparse connections outperform baseline channel reduction.

Significant computational and memory savings achieved.

Effective in large networks like VGG and Inception V3.

Abstract

Deep convolutional networks are well-known for their high computational and memory demands. Given limited resources, how does one design a network that balances its size, training time, and prediction accuracy? A surprisingly effective approach to trade accuracy for size and speed is to simply reduce the number of channels in each convolutional layer by a fixed fraction and retrain the network. In many cases this leads to significantly smaller networks with only minimal changes to accuracy. In this paper, we take a step further by empirically examining a strategy for deactivating connections between filters in convolutional layers in a way that allows us to harvest savings both in run-time and memory for many network architectures. More specifically, we generalize 2D convolution to use a channel-wise sparse connection structure and show that this leads to significantly better results than the baseline approach for large networks including VGG and Inception V3.