STanHop: Sparse Tandem Hopfield Model for Memory-Enhanced Time Series Prediction

TL;DR

STanHop-Net is a memory-augmented neural network for multivariate time series prediction that uses sparse Hopfield layers and external memory modules for improved performance and rapid response to sudden events.

Contribution

The paper introduces STanHop, a novel sparse Hopfield-based neural network block, and integrates it into a hierarchical framework with external memory modules for enhanced time series prediction.

Findings

Effective in synthetic and real-world datasets

Outperforms existing models in memory retrieval accuracy

Responds swiftly to sudden events

Abstract

We present STanHop-Net (Sparse Tandem Hopfield Network) for multivariate time series prediction with memory-enhanced capabilities. At the heart of our approach is STanHop, a novel Hopfield-based neural network block, which sparsely learns and stores both temporal and cross-series representations in a data-dependent fashion. In essence, STanHop sequentially learn temporal representation and cross-series representation using two tandem sparse Hopfield layers. In addition, StanHop incorporates two additional external memory modules: a Plug-and-Play module and a Tune-and-Play module for train-less and task-aware memory-enhancements, respectively. They allow StanHop-Net to swiftly respond to certain sudden events. Methodologically, we construct the StanHop-Net by stacking STanHop blocks in a hierarchical fashion, enabling multi-resolution feature extraction with resolution-specific sparsity. Theoretically, we introduce a sparse extension of the modern Hopfield model (Generalized Sparse Modern Hopfield Model) and show that it endows a tighter memory retrieval error compared to the dense counterpart without sacrificing memory capacity. Empirically, we validate the efficacy of our framework on both synthetic and real-world settings.