How to train accurate BNNs for embedded systems?

TL;DR

This paper reviews various repair methods to improve the accuracy of binary neural networks (BNNs) for embedded systems, analyzing their effectiveness and discussing future research directions.

Contribution

It provides an empirical overview of repair techniques for BNNs, categorizes them, and evaluates their benefits on benchmark datasets, highlighting promising approaches.

Findings

Feature binarizer, feature normalization, and double residual are most beneficial repair methods.

Progress has been made in reducing the accuracy gap of BNNs.

Trade-offs between accuracy improvements and energy costs are discussed.

Abstract

A key enabler of deploying convolutional neural networks on resource-constrained embedded systems is the binary neural network (BNN). BNNs save on memory and simplify computation by binarizing both features and weights. Unfortunately, binarization is inevitably accompanied by a severe decrease in accuracy. To reduce the accuracy gap between binary and full-precision networks, many repair methods have been proposed in the recent past, which we have classified and put into a single overview in this chapter. The repair methods are divided into two main branches, training techniques and network topology changes, which can further be split into smaller categories. The latter category introduces additional cost (energy consumption or additional area) for an embedded system, while the former does not. From our overview, we observe that progress has been made in reducing the accuracy gap, but BNN papers are not aligned on what repair methods should be used to get highly accurate BNNs. Therefore, this chapter contains an empirical review that evaluates the benefits of many repair methods in isolation over the ResNet-20\&CIFAR10 and ResNet-18\&CIFAR100 benchmarks. We found three repair categories most beneficial: feature binarizer, feature normalization, and double residual. Based on this review we discuss future directions and research opportunities. We sketch the benefit and costs associated with BNNs on embedded systems because it remains to be seen whether BNNs will be able to close the accuracy gap while staying highly energy-efficient on resource-constrained embedded systems.