Energy-based General Sequential Episodic Memory Networks at the Adiabatic Limit

TL;DR

This paper introduces a new class of sequential episodic memory models with dynamic energy surfaces, enabling the storage and retrieval of ordered memories, inspired by neuroscience and implemented with novel learning rules.

Contribution

The paper proposes GSEMM, a new theoretical framework for sequential episodic memories with adiabatic energy surfaces, asymmetric synapses, and online learning rules, bridging neuroscience and AI.

Findings

DSEM's storage capacity grows exponentially with neurons.

The energy minimization learning rule effectively learns sequences.

Models exhibit stable sequential memory retrieval.

Abstract

The General Associative Memory Model (GAMM) has a constant state-dependant energy surface that leads the output dynamics to fixed points, retrieving single memories from a collection of memories that can be asynchronously preloaded. We introduce a new class of General Sequential Episodic Memory Models (GSEMM) that, in the adiabatic limit, exhibit temporally changing energy surface, leading to a series of meta-stable states that are sequential episodic memories. The dynamic energy surface is enabled by newly introduced asymmetric synapses with signal propagation delays in the network's hidden layer. We study the theoretical and empirical properties of two memory models from the GSEMM class, differing in their activation functions. LISEM has non-linearities in the feature layer, whereas DSEM has non-linearity in the hidden layer. In principle, DSEM has a storage capacity that grows exponentially with the number of neurons in the network. We introduce a learning rule for the synapses based on the energy minimization principle and show it can learn single memories and their sequential relationships online. This rule is similar to the Hebbian learning algorithm and Spike-Timing Dependent Plasticity (STDP), which describe conditions under which synapses between neurons change strength. Thus, GSEMM combines the static and dynamic properties of episodic memory under a single theoretical framework and bridges neuroscience, machine learning, and artificial intelligence.