Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units

TL;DR

This paper leverages the high sparsity of ReLU activations in deep convolutional networks to accelerate convolution computations by skipping zero-valued neurons, achieving notable speedups on CPUs.

Contribution

It introduces a sparse convolution algorithm that exploits ReLU sparsity to improve computational efficiency in deep CNNs.

Findings

Achieves speedup when ReLU sparsity exceeds 0.9

Demonstrates efficiency gains on CPU architectures

Validates the approach on popular deep CNN architectures

Abstract

Rectifier neuron units (ReLUs) have been widely used in deep convolutional networks. An ReLU converts negative values to zeros, and does not change positive values, which leads to a high sparsity of neurons. In this work, we first examine the sparsity of the outputs of ReLUs in some popular deep convolutional architectures. And then we use the sparsity property of ReLUs to accelerate the calculation of convolution by skipping calculations of zero-valued neurons. The proposed sparse convolution algorithm achieves some speedup improvements on CPUs compared to the traditional matrix-matrix multiplication algorithm for convolution when the sparsity is not less than 0.9.