On-Device Learning with Binary Neural Networks

TL;DR

This paper introduces a novel on-device continual learning method using Binary Neural Networks, optimizing for low-power embedded devices by hybrid quantization and demonstrating its effectiveness through experiments.

Contribution

It presents the first on-device continual learning approach with Binary Neural Networks, combining hybrid quantization for improved efficiency and precision.

Findings

Effective on-device learning demonstrated with BNNs

Hybrid quantization improves gradient update precision

Method suitable for low-power embedded devices

Abstract

Existing Continual Learning (CL) solutions only partially address the constraints on power, memory and computation of the deep learning models when deployed on low-power embedded CPUs. In this paper, we propose a CL solution that embraces the recent advancements in CL field and the efficiency of the Binary Neural Networks (BNN), that use 1-bit for weights and activations to efficiently execute deep learning models. We propose a hybrid quantization of CWR* (an effective CL approach) that considers differently forward and backward pass in order to retain more precision during gradient update step and at the same time minimizing the latency overhead. The choice of a binary network as backbone is essential to meet the constraints of low power devices and, to the best of authors' knowledge, this is the first attempt to prove on-device learning with BNN. The experimental validation carried out confirms the validity and the suitability of the proposed method.