Learning with Local Gradients at the Edge

TL;DR

This paper introduces tpSGD, a novel backpropagation-free training algorithm for neural networks that reduces memory usage and enables efficient learning on edge devices, achieving comparable accuracy to traditional methods.

Contribution

The paper presents tpSGD, a new gradient-free optimization method that generalizes target projection for training various neural network architectures with minimal memory.

Findings

tpSGD performs within 5% accuracy of backpropagation on shallow networks.

Outperforms other gradient-free algorithms in accuracy and efficiency.

Enables training of deep networks like VGG with reduced memory requirements.

Abstract

To enable learning on edge devices with fast convergence and low memory, we present a novel backpropagation-free optimization algorithm dubbed Target Projection Stochastic Gradient Descent (tpSGD). tpSGD generalizes direct random target projection to work with arbitrary loss functions and extends target projection for training recurrent neural networks (RNNs) in addition to feedforward networks. tpSGD uses layer-wise stochastic gradient descent (SGD) and local targets generated via random projections of the labels to train the network layer-by-layer with only forward passes. tpSGD doesn't require retaining gradients during optimization, greatly reducing memory allocation compared to SGD backpropagation (BP) methods that require multiple instances of the entire neural network weights, input/output, and intermediate results. Our method performs comparably to BP gradient-descent within 5% accuracy on relatively shallow networks of fully connected layers, convolutional layers, and recurrent layers. tpSGD also outperforms other state-of-the-art gradient-free algorithms in shallow models consisting of multi-layer perceptrons, convolutional neural networks (CNNs), and RNNs with competitive accuracy and less memory and time. We evaluate the performance of tpSGD in training deep neural networks (e.g. VGG) and extend the approach to multi-layer RNNs. These experiments highlight new research directions related to optimized layer-based adaptor training for domain-shift using tpSGD at the edge.