Modern Hopfield Networks with Continuous-Time Memories

TL;DR

This paper introduces a continuous-time memory approach for Hopfield networks inspired by human working memory, enabling efficient storage and reduced computational costs while maintaining competitive performance.

Contribution

It proposes a novel continuous-time memory model for Hopfield networks that compresses large memories and improves efficiency, inspired by psychological theories of working memory.

Findings

Maintains competitive performance with traditional Hopfield networks.

Reduces computational costs through memory compression.

Aligns neural network models with human cognitive theories.

Abstract

Recent research has established a connection between modern Hopfield networks (HNs) and transformer attention heads, with guarantees of exponential storage capacity. However, these models still face challenges scaling storage efficiently. Inspired by psychological theories of continuous neural resource allocation in working memory, we propose an approach that compresses large discrete Hopfield memories into smaller, continuous-time memories. Leveraging continuous attention, our new energy function modifies the update rule of HNs, replacing the traditional softmax-based probability mass function with a probability density, over the continuous memory. This formulation aligns with modern perspectives on human executive function, offering a principled link between attractor dynamics in working memory and resource-efficient memory allocation. Our framework maintains competitive performance with HNs while leveraging a compressed memory, reducing computational costs across synthetic and video datasets.