Towards training digitally-tied analog blocks via hybrid gradient computation

TL;DR

This paper introduces a hybrid gradient computation method for energy-based models combining analog and digital components, enabling scalable training of neural networks with improved efficiency and performance.

Contribution

It proposes Feedforward-tied Energy-based Models (ff-EBMs) and a novel gradient algorithm that integrates analog and digital circuits for neural network training.

Findings

Achieved new state-of-the-art top-1 accuracy of 46% on ImageNet32 with EP.

Demonstrated flexible splitting of Deep Hopfield Networks without performance loss.

Validated the effectiveness of hybrid gradient computation in scalable neural architectures.

Abstract

Power efficiency is plateauing in the standard digital electronics realm such that novel hardware, models, and algorithms are needed to reduce the costs of AI training. The combination of energy-based analog circuits and the Equilibrium Propagation (EP) algorithm constitutes one compelling alternative compute paradigm for gradient-based optimization of neural nets. Existing analog hardware accelerators, however, typically incorporate digital circuitry to sustain auxiliary non-weight-stationary operations, mitigate analog device imperfections, and leverage existing digital accelerators.This heterogeneous hardware approach calls for a new theoretical model building block. In this work, we introduce Feedforward-tied Energy-based Models (ff-EBMs), a hybrid model comprising feedforward and energy-based blocks accounting for digital and analog circuits. We derive a novel algorithm to compute gradients end-to-end in ff-EBMs by backpropagating and "eq-propagating" through feedforward and energy-based parts respectively, enabling EP to be applied to much more flexible and realistic architectures. We experimentally demonstrate the effectiveness of the proposed approach on ff-EBMs where Deep Hopfield Networks (DHNs) are used as energy-based blocks. We first show that a standard DHN can be arbitrarily split into any uniform size while maintaining performance. We then train ff-EBMs on ImageNet32 where we establish new SOTA performance in the EP literature (46 top-1 %). Our approach offers a principled, scalable, and incremental roadmap to gradually integrate self-trainable analog computational primitives into existing digital accelerators.