Scalable Equilibrium Propagation via Intermediate Error Signals for Deep Convolutional CRNNs

TL;DR

This paper introduces a novel equilibrium propagation framework with intermediate error signals, enabling scalable training of deep convolutional CRNNs and achieving state-of-the-art results on CIFAR datasets.

Contribution

It is the first to incorporate layer-wise learning signals and knowledge distillation into EP, allowing training of much deeper neural networks.

Findings

Achieved state-of-the-art performance on CIFAR-10 and CIFAR-100.

Demonstrated scalability of EP to deep VGG architectures.

Enhanced convergence of neuron dynamics in deep networks.

Abstract

Equilibrium Propagation (EP) is a biologically inspired local learning rule first proposed for convergent recurrent neural networks (CRNNs), in which synaptic updates depend only on neuron states from two distinct phases. EP estimates gradients that closely align with those computed by Backpropagation Through Time (BPTT) while significantly reducing computational demands, positioning it as a potential candidate for on-chip training in neuromorphic architectures. However, prior studies on EP have been constrained to shallow architectures, as deeper networks suffer from the vanishing gradient problem, leading to convergence difficulties in both energy minimization and gradient computation. To alleviate the vanishing gradient problem in deep EP networks, we propose a novel EP framework that incorporates layer-wise learning signals to provide auxiliary supervision, which enhances the convergence of neuron dynamics. This is the first work to integrate knowledge distillation and local error signals into EP, enabling the training of significantly deeper architectures. Our proposed approach achieves state-of-the-art performance on the CIFAR-10 and CIFAR-100 datasets, showcasing its scalability on deep VGG architectures. These results represent a significant advancement in the scalability of EP, suggesting that intermediate learning signals can extend the practical applicability of EP to deeper architectures.