Hybrid Binary Networks: Optimizing for Accuracy, Efficiency and Memory

TL;DR

This paper proposes a layer-wise selective binarization method for neural networks, improving accuracy while maintaining efficiency and memory benefits, demonstrated on ImageNet and sketch datasets.

Contribution

It introduces an algorithm to select which layer inputs to binarize, balancing accuracy and computational cost, and applies it to improve binarized network performance.

Findings

3-4% accuracy improvement on ImageNet with minimal speed impact

Over 8% accuracy increase on sketch datasets like TU-Berlin

Enables binarization of last layer weights for better accuracy

Abstract

Binarization is an extreme network compression approach that provides large computational speedups along with energy and memory savings, albeit at significant accuracy costs. We investigate the question of where to binarize inputs at layer-level granularity and show that selectively binarizing the inputs to specific layers in the network could lead to significant improvements in accuracy while preserving most of the advantages of binarization. We analyze the binarization tradeoff using a metric that jointly models the input binarization-error and computational cost and introduce an efficient algorithm to select layers whose inputs are to be binarized. Practical guidelines based on insights obtained from applying the algorithm to a variety of models are discussed. Experiments on Imagenet dataset using AlexNet and ResNet-18 models show 3-4% improvements in accuracy over fully binarized networks with minimal impact on compression and computational speed. The improvements are even more substantial on sketch datasets like TU-Berlin, where we match state-of-the-art accuracy as well, getting over 8% increase in accuracies. We further show that our approach can be applied in tandem with other forms of compression that deal with individual layers or overall model compression (e.g., SqueezeNets). Unlike previous quantization approaches, we are able to binarize the weights in the last layers of a network, which often have a large number of parameters, resulting in significant improvement in accuracy over fully binarized models.