Low-Variance Forward Gradients using Direct Feedback Alignment and Momentum

TL;DR

This paper introduces a novel forward gradient method combining direct feedback alignment and momentum, reducing variance and enabling faster, more efficient learning suitable for neuromorphic hardware.

Contribution

It proposes the Forward Direct Feedback Alignment algorithm, which lowers variance in forward gradients, improving convergence and performance over existing local learning methods.

Findings

Achieves lower variance than previous forward gradient techniques.

Enables faster convergence and better performance in neural network training.

Supports online learning compatible with neuromorphic systems.

Abstract

Supervised learning in deep neural networks is commonly performed using error backpropagation. However, the sequential propagation of errors during the backward pass limits its scalability and applicability to low-powered neuromorphic hardware. Therefore, there is growing interest in finding local alternatives to backpropagation. Recently proposed methods based on forward-mode automatic differentiation suffer from high variance in large deep neural networks, which affects convergence. In this paper, we propose the Forward Direct Feedback Alignment algorithm that combines Activity-Perturbed Forward Gradients with Direct Feedback Alignment and momentum. We provide both theoretical proofs and empirical evidence that our proposed method achieves lower variance than forward gradient techniques. In this way, our approach enables faster convergence and better performance when compared to other local alternatives to backpropagation and opens a new perspective for the development of online learning algorithms compatible with neuromorphic systems.