A Gradient Estimator for Time-Varying Electrical Networks with Non-Linear Dissipation

TL;DR

This paper introduces a local gradient estimator for time-varying neural networks modeled as electrical circuits with nonlinear and dissipative elements, enabling more biologically plausible learning.

Contribution

It extends equilibrium propagation to dynamic, directed neural networks using electrical circuit models with nonlinear components and derives a local gradient estimator for physical parameters.

Findings

Derived a Lagrangian for nonlinear electrical circuit models of neural networks.

Developed a local gradient estimator based on circuit theory.

Suggested extensions to biologically plausible neuron models.

Abstract

We propose a method for extending the technique of equilibrium propagation for estimating gradients in fixed-point neural networks to the more general setting of directed, time-varying neural networks by modeling them as electrical circuits. We use electrical circuit theory to construct a Lagrangian capable of describing deep, directed neural networks modeled using nonlinear capacitors and inductors, linear resistors and sources, and a special class of nonlinear dissipative elements called fractional memristors. We then derive an estimator for the gradient of the physical parameters of the network, such as synapse conductances, with respect to an arbitrary loss function. This estimator is entirely local, in that it only depends on information locally available to each synapse. We conclude by suggesting methods for extending these results to networks of biologically plausible neurons, e.g. Hodgkin-Huxley neurons.