Sparsifying Binary Networks

TL;DR

This paper introduces sparse binary neural networks (SBNNs), which incorporate sparsity and a new quantization method to significantly enhance compression and efficiency of binary neural networks for resource-constrained IoT devices.

Contribution

The paper proposes a novel SBNN model and training scheme that introduces sparsity and a new weight binarization function, enabling higher compression and efficiency without losing accuracy.

Findings

SBNNs achieve high compression rates.

SBNNs reduce the number of operations at inference.

SBNNs maintain generalization performance.

Abstract

Binary neural networks (BNNs) have demonstrated their ability to solve complex tasks with comparable accuracy as full-precision deep neural networks (DNNs), while also reducing computational power and storage requirements and increasing the processing speed. These properties make them an attractive alternative for the development and deployment of DNN-based applications in Internet-of-Things (IoT) devices. Despite the recent improvements, they suffer from a fixed and limited compression factor that may result insufficient for certain devices with very limited resources. In this work, we propose sparse binary neural networks (SBNNs), a novel model and training scheme which introduces sparsity in BNNs and a new quantization function for binarizing the network's weights. The proposed SBNN is able to achieve high compression factors and it reduces the number of operations and parameters at inference time. We also provide tools to assist the SBNN design, while respecting hardware resource constraints. We study the generalization properties of our method for different compression factors through a set of experiments on linear and convolutional networks on three datasets. Our experiments confirm that SBNNs can achieve high compression rates, without compromising generalization, while further reducing the operations of BNNs, making SBNNs a viable option for deploying DNNs in cheap, low-cost, limited-resources IoT devices and sensors.