Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing its Gradient Estimator Bias

TL;DR

This paper identifies and reduces the bias in Equilibrium Propagation's gradient estimator, enabling it to scale to deep convolutional networks and making it a viable, energy-efficient alternative to backpropagation for complex visual tasks.

Contribution

The authors demonstrate that reducing the bias in EP's gradient estimate allows training deep ConvNets, extending EP's applicability beyond simple datasets like MNIST.

Findings

Bias in EP's gradient estimate hinders deep network training.

Cancelling the bias enables training of deep ConvNets with EP.

EP becomes a scalable method for error gradient computation in deep neural networks.

Abstract

Equilibrium Propagation (EP) is a biologically-inspired counterpart of Backpropagation Through Time (BPTT) which, owing to its strong theoretical guarantees and the locality in space of its learning rule, fosters the design of energy-efficient hardware dedicated to learning. In practice, however, EP does not scale to visual tasks harder than MNIST. In this work, we show that a bias in the gradient estimate of EP, inherent in the use of finite nudging, is responsible for this phenomenon and that cancelling it allows training deep ConvNets by EP, including architectures with distinct forward and backward connections. These results highlight EP as a scalable approach to compute error gradients in deep neural networks, thereby motivating its hardware implementation.