Long-Sequence Memory with Temporal Kernels and Dense Hopfield Functionals

TL;DR

This paper introduces a novel energy functional with temporal kernels for long-sequence memory, enhancing the capacity and efficiency of dense Hopfield networks for sequential pattern retrieval, with applications in transformers and time-series analysis.

Contribution

It proposes a new temporal kernel-based approach to extend dense Hopfield networks for long-sequence memory and sequential retrieval, improving long-term dependency handling.

Findings

Successful retrieval of high-dimensional movie frames over extended sequences

Enhanced long-sequence modeling in transformer architectures

Potential improvements in natural language processing and forecasting

Abstract

In this study we introduce a novel energy functional for long-sequence memory, building upon the framework of dense Hopfield networks which achieves exponential storage capacity through higher-order interactions. Building upon earlier work on long-sequence Hopfield memory models, we propose a temporal kernal $K(m, k)$ to incorporate temporal dependencies, enabling efficient sequential retrieval of patterns over extended sequences. We demonstrate the successful application of this technique for the storage and sequential retrieval of movies frames which are well suited for this because of the high dimensional vectors that make up each frame creating enough variation between even sequential frames in the high dimensional space. The technique has applications in modern transformer architectures, including efficient long-sequence modeling, memory augmentation, improved attention with temporal bias, and enhanced handling of long-term dependencies in time-series data. Our model offers a promising approach to address the limitations of transformers in long-context tasks, with potential implications for natural language processing, forecasting, and beyond.