Feed-Forward Optimization With Delayed Feedback for Neural Network Training

TL;DR

This paper introduces F$^3$, a biologically plausible neural network training method using approximate gradients with delayed feedback, significantly improving performance over previous approaches and narrowing the gap with backpropagation.

Contribution

F$^3$ is a novel training algorithm that employs fixed random feedback and delayed error signals to enhance biological plausibility while maintaining high predictive accuracy.

Findings

F$^3$ improves classification accuracy by up to 56% compared to similar methods.

F$^3$ achieves up to 96% of backpropagation's performance on regression tasks.

The approach is effective across various architectures, including Transformers.

Abstract

Backpropagation has long been criticized for being biologically implausible due to its reliance on concepts that are not viable in natural learning processes. Two core issues are the weight transport and update locking problems caused by the forward-backward dependencies, which limit biological plausibility, computational efficiency, and parallelization. Although several alternatives have been proposed to increase biological plausibility, they often come at the cost of reduced predictive performance. This paper proposes an alternative approach to training feed-forward neural networks addressing these issues by using approximate gradient information. We introduce Feed-Forward with delayed Feedback (F$^3$), which approximates gradients using fixed random feedback paths and delayed error information from the previous epoch to balance biological plausibility with predictive performance. We evaluate F$^3$ across multiple tasks and architectures, including both fully-connected and Transformer networks. Our results demonstrate that, compared to similarly plausible approaches, F$^3$ significantly improves predictive performance, narrowing the gap to backpropagation by up to 56% for classification and 96% for regression. This work is a step towards more biologically plausible learning algorithms while opening up new avenues for energy-efficient and parallelizable neural network training.