Training End-to-End Analog Neural Networks with Equilibrium Propagation

TL;DR

This paper presents a method for training analog neural networks using equilibrium propagation, leveraging their energy-based properties to enable efficient, on-chip learning with resistive devices, demonstrated on MNIST classification.

Contribution

It introduces a novel training approach for analog neural networks based on energy functions and equilibrium propagation, suitable for resistive hardware implementations.

Findings

Achieved MNIST classification performance comparable or superior to software neural networks.

Demonstrated that conductance updates can be computed locally from voltage drops.

Validated the approach using SPICE simulations of electrical circuit dynamics.

Abstract

We introduce a principled method to train end-to-end analog neural networks by stochastic gradient descent. In these analog neural networks, the weights to be adjusted are implemented by the conductances of programmable resistive devices such as memristors [Chua, 1971], and the nonlinear transfer functions (or `activation functions') are implemented by nonlinear components such as diodes. We show mathematically that a class of analog neural networks (called nonlinear resistive networks) are energy-based models: they possess an energy function as a consequence of Kirchhoff's laws governing electrical circuits. This property enables us to train them using the Equilibrium Propagation framework [Scellier and Bengio, 2017]. Our update rule for each conductance, which is local and relies solely on the voltage drop across the corresponding resistor, is shown to compute the gradient of the loss function. Our numerical simulations, which use the SPICE-based Spectre simulation framework to simulate the dynamics of electrical circuits, demonstrate training on the MNIST classification task, performing comparably or better than equivalent-size software-based neural networks. Our work can guide the development of a new generation of ultra-fast, compact and low-power neural networks supporting on-chip learning.