Nonequilibrium Thermodynamics of Associative Memory Continuous-Time Recurrent Neural Networks

TL;DR

This paper introduces a novel class of continuous-time recurrent neural networks based on asymmetric associative memories, providing a thermodynamic framework to interpret their complex, nonequilibrium dynamics and improve interpretability.

Contribution

It develops a mathematical formalism for nonequilibrium thermodynamics in associative CTRNNs, enabling direct computation of macroscopic observables and entropy measures.

Findings

The model captures fluctuations in nonequilibrium dynamics.

It allows for direct calculation of entropy and entropy dissipation.

The approach enhances interpretability of complex sequence-encoding networks.

Abstract

Continuous-Time Recurrent Neural Networks (CTRNNs) have been widely used for their capacity to model complex temporal behaviour. However, their internal dynamics often remain difficult to interpret. In this paper, we propose a new class of CTRNNs based on Hopfield-like associative memories with asymmetric couplings. This model combines the expressive power of associative memories with a tractable mathematical formalism to characterize fluctuations in nonequilibrium dynamics. We show that this mathematical description allows us to directly compute the evolution of its macroscopic observables (the encoded features), as well as the instantaneous entropy and entropy dissipation of the system, thereby offering a bridge between dynamical systems descriptions of low-dimensional observables and the statistical mechanics of large nonequilibrium networks. Our results suggest that these nonequilibrium associative CTRNNs can serve as more interpretable models for complex sequence-encoding networks.