Efficient Training Convolutional Neural Networks on Edge Devices with Gradient-pruned Sign-symmetric Feedback Alignment

TL;DR

This paper introduces a novel training method for convolutional neural networks on edge devices that significantly improves energy efficiency with minimal accuracy loss, enabling more practical distributed learning.

Contribution

It proposes a gradient-pruned sign-symmetric feedback alignment method that leverages redundancy in backpropagation to enhance energy efficiency on edge devices.

Findings

5x energy efficiency improvement over prior methods

Negligible accuracy loss in CNN training

Effective training on resource-constrained edge devices

Abstract

With the prosperity of mobile devices, the distributed learning approach enabling model training with decentralized data has attracted wide research. However, the lack of training capability for edge devices significantly limits the energy efficiency of distributed learning in real life. This paper describes a novel approach of training DNNs exploiting the redundancy and the weight asymmetry potential of conventional backpropagation. We demonstrate that with negligible classification accuracy loss, the proposed approach outperforms the prior arts by 5x in terms of energy efficiency.