Biologically Motivated Algorithms for Propagating Local Target Representations

TL;DR

This paper introduces biologically inspired learning algorithms, LRA-E and an improved Difference Target Propagation, that achieve stable and strong performance in training deep neural networks, offering plausible alternatives to back-propagation.

Contribution

The paper proposes two novel biologically motivated algorithms, LRA-E and an improved Difference Target Propagation, demonstrating their effectiveness in deep network training.

Findings

LRA-E outperforms other algorithms in stability and generalization.

Both algorithms perform well on benchmarks with deep, nonlinear networks.

They provide plausible alternatives to traditional back-propagation.

Abstract

Finding biologically plausible alternatives to back-propagation of errors is a fundamentally important challenge in artificial neural network research. In this paper, we propose a learning algorithm called error-driven Local Representation Alignment (LRA-E), which has strong connections to predictive coding, a theory that offers a mechanistic way of describing neurocomputational machinery. In addition, we propose an improved variant of Difference Target Propagation, another procedure that comes from the same family of algorithms as LRA-E. We compare our procedures to several other biologically-motivated algorithms, including two feedback alignment algorithms and Equilibrium Propagation. In two benchmarks, we find that both of our proposed algorithms yield stable performance and strong generalization compared to other competing back-propagation alternatives when training deeper, highly nonlinear networks, with LRA-E performing the best overall.