Toward Practical Equilibrium Propagation: Brain-inspired Recurrent Neural Network with Feedback Regulation and Residual Connections

TL;DR

This paper introduces a biologically inspired recurrent neural network with feedback regulation and residual connections, significantly improving the efficiency and stability of Equilibrium Propagation for large-scale brain-like AI systems.

Contribution

It proposes a novel feedback-regulated residual RNN that enhances EP convergence and reduces computational costs, making brain-inspired learning more practical.

Findings

EP performance on par with backpropagation in benchmark tasks

Feedback regulation reduces spectral radius and accelerates convergence

Residual connections alleviate vanishing gradients in deep RNNs

Abstract

Brain-like intelligent systems need brain-like learning methods. Equilibrium Propagation (EP) is a biologically plausible learning framework with strong potential for brain-inspired computing hardware. However, existing im-plementations of EP suffer from instability and prohibi-tively high computational costs. Inspired by the structure and dynamics of the brain, we propose a biologically plau-sible Feedback-regulated REsidual recurrent neural network (FRE-RNN) and study its learning performance in EP framework. Feedback regulation enables rapid convergence by reducing the spectral radius. The improvement in con-vergence property reduces the computational cost and train-ing time of EP by orders of magnitude, delivering perfor-mance on par with backpropagation (BP) in benchmark tasks. Meanwhile, residual connections with brain-inspired topologies help alleviate the vanishing gradient problem that arises when feedback pathways are weak in deep RNNs. Our approach substantially enhances the applicabil-ity and practicality of EP in large-scale networks that un-derpin artificial intelligence. The techniques developed here also offer guidance to implementing in-situ learning in physical neural networks.