Low-complexity Approximate Convolutional Neural Networks

TL;DR

This paper introduces a method to significantly reduce the computational complexity of convolutional neural networks by approximating their filters with binary matrices, enabling efficient, multiplication-free operations suitable for low-power hardware.

Contribution

The authors propose a novel binary linear programming approach to approximate ConvNet filters, drastically reducing computational complexity while preserving performance across various network sizes.

Findings

Low-complexity approximations maintain near-original accuracy.

Method applicable to diverse ConvNet architectures.

Enables efficient hardware implementations with addition and bit-shifting.

Abstract

In this paper, we present an approach for minimizing the computational complexity of trained Convolutional Neural Networks (ConvNet). The idea is to approximate all elements of a given ConvNet and replace the original convolutional filters and parameters (pooling and bias coefficients; and activation function) with efficient approximations capable of extreme reductions in computational complexity. Low-complexity convolution filters are obtained through a binary (zero-one) linear programming scheme based on the Frobenius norm over sets of dyadic rationals. The resulting matrices allow for multiplication-free computations requiring only addition and bit-shifting operations. Such low-complexity structures pave the way for low-power, efficient hardware designs. We applied our approach on three use cases of different complexity: (i) a "light" but efficient ConvNet for face detection (with around 1000 parameters); (ii) another one for hand-written digit classification (with more than 180000 parameters); and (iii) a significantly larger ConvNet: AlexNet with $\approx$1.2 million matrices. We evaluated the overall performance on the respective tasks for different levels of approximations. In all considered applications, very low-complexity approximations have been derived maintaining an almost equal classification performance.