Sequence Learning Using Equilibrium Propagation

TL;DR

This paper advances equilibrium propagation (EP) for sequence classification by integrating modern hopfield networks as attention mechanisms, enabling EP to handle dynamic input sequences in NLP tasks like sentiment analysis and natural language inference.

Contribution

It introduces a novel approach combining EP with modern hopfield networks to enable sequence learning, overcoming previous static input limitations of EP.

Findings

Successfully applied EP to NLP sequence tasks

Extended EP applicability to dynamic input sequences

Demonstrated competitive results on sentiment analysis and NLI datasets

Abstract

Equilibrium Propagation (EP) is a powerful and more bio-plausible alternative to conventional learning frameworks such as backpropagation. The effectiveness of EP stems from the fact that it relies only on local computations and requires solely one kind of computational unit during both of its training phases, thereby enabling greater applicability in domains such as bio-inspired neuromorphic computing. The dynamics of the model in EP is governed by an energy function and the internal states of the model consequently converge to a steady state following the state transition rules defined by the same. However, by definition, EP requires the input to the model (a convergent RNN) to be static in both the phases of training. Thus it is not possible to design a model for sequence classification using EP with an LSTM or GRU like architecture. In this paper, we leverage recent developments in modern hopfield networks to further understand energy based models and develop solutions for complex sequence classification tasks using EP while satisfying its convergence criteria and maintaining its theoretical similarities with recurrent backpropagation. We explore the possibility of integrating modern hopfield networks as an attention mechanism with convergent RNN models used in EP, thereby extending its applicability for the first time on two different sequence classification tasks in natural language processing viz. sentiment analysis (IMDB dataset) and natural language inference (SNLI dataset).