End-to-end fully-binarized network design: from Generic Learned Thermometer to Block Pruning

TL;DR

This paper introduces a novel input encoding method called Generic Learned Thermometer (GLT) for Binary Neural Networks, combined with block pruning and knowledge distillation to create lightweight, fully-binarized models suitable for in-sensor inference.

Contribution

It proposes the GLT encoding technique for improved input data representation and combines it with block pruning and knowledge distillation for compact BNNs.

Findings

GLT improves accuracy by intrinsic global tone mapping.

Combined techniques produce models under 1MB with minimal accuracy loss.

The approach is effective for in-sensor always-on inference applications.

Abstract

Existing works on Binary Neural Network (BNN) mainly focus on model's weights and activations while discarding considerations on the input raw data. This article introduces Generic Learned Thermometer (GLT), an encoding technique to improve input data representation for BNN, relying on learning non linear quantization thresholds. This technique consists in multiple data binarizations which can advantageously replace a conventional Analog to Digital Conversion (ADC) that uses natural binary coding. Additionally, we jointly propose a compact topology with light-weight grouped convolutions being trained thanks to block pruning and Knowledge Distillation (KD), aiming at reducing furthermore the model size so as its computational complexity. We show that GLT brings versatility to the BNN by intrinsically performing global tone mapping, enabling significant accuracy gains in practice (demonstrated by simulations on the STL-10 and VWW datasets). Moreover, when combining GLT with our proposed block-pruning technique, we successfully achieve lightweight (under 1Mb), fully-binarized models with limited accuracy degradation while being suitable for in-sensor always-on inference use cases.