Exponential Dynamic Energy Network for High Capacity Sequence Memory

TL;DR

EDEN introduces a novel energy-based neural architecture that significantly enhances sequence memory capacity by evolving energy functions over multiple timescales, outperforming traditional models and aligning with biological observations.

Contribution

The paper proposes EDEN, a dynamic energy network that extends static energy models to temporal domains, achieving exponential sequence memory capacity and providing analytical insights into its dynamics.

Findings

EDEN achieves exponential sequence memory capacity $O( ext{ extgamma}^N)$.

EDEN's dynamics resemble activity of memory-related brain cells.

Analytical derivation of memory escape times and phase transition analysis.

Abstract

The energy paradigm, exemplified by Hopfield networks, offers a principled framework for memory in neural systems by interpreting dynamics as descent on an energy surface. While powerful for static associative memories, it falls short in modeling sequential memory, where transitions between memories are essential. We introduce the Exponential Dynamic Energy Network (EDEN), a novel architecture that extends the energy paradigm to temporal domains by evolving the energy function over multiple timescales. EDEN combines a static high-capacity energy network with a slow, asymmetrically interacting modulatory population, enabling robust and controlled memory transitions. We formally derive short-timescale energy functions that govern local dynamics and use them to analytically compute memory escape times, revealing a phase transition between static and dynamic regimes. The analysis of capacity, defined as the number of memories that can be stored with minimal error rate as a function of the dimensions of the state space (number of feature neurons), for EDEN shows that it achieves exponential sequence memory capacity $O(\gamma^N)$, outperforming the linear capacity $O(N)$ of conventional models. Furthermore, EDEN's dynamics resemble the activity of time and ramping cells observed in the human brain during episodic memory tasks, grounding its biological relevance. By unifying static and sequential memory within a dynamic energy framework, EDEN offers a scalable and interpretable model for high-capacity temporal memory in both artificial and biological systems.