Fast ConvNets Using Group-wise Brain Damage

TL;DR

This paper introduces a group-wise brain damage method that prunes convolutional kernels with regularization, enabling faster convolutional neural networks by reducing convolutions to efficient matrix multiplications.

Contribution

The paper proposes a novel group-wise pruning technique using regularization to accelerate convolutional layers in neural networks.

Findings

Achieves competitive speedup on AlexNet

Pruning reduces convolutional complexity

Maintains accuracy with group-wise regularization

Abstract

We revisit the idea of brain damage, i.e. the pruning of the coefficients of a neural network, and suggest how brain damage can be modified and used to speedup convolutional layers. The approach uses the fact that many efficient implementations reduce generalized convolutions to matrix multiplications. The suggested brain damage process prunes the convolutional kernel tensor in a group-wise fashion by adding group-sparsity regularization to the standard training process. After such group-wise pruning, convolutions can be reduced to multiplications of thinned dense matrices, which leads to speedup. In the comparison on AlexNet, the method achieves very competitive performance.