Hidden Traveling Waves bind Working Memory Variables in Recurrent Neural Networks

TL;DR

This paper introduces a novel neural working memory model based on traveling wave dynamics, demonstrating its ability to store information, improve learning, and relate to AI architectures like RNNs and transformers.

Contribution

The study presents a new wave-based neural memory model that diverges from static storage, showing how traveling waves can enhance learning and information retention in neural networks.

Findings

The wave model reliably stores external information.

It improves learning by mitigating the diminishing gradient problem.

The linear boundary condition model aligns with existing RNN state space matrices.

Abstract

Traveling waves are a fundamental phenomenon in the brain, playing a crucial role in short-term information storage. In this study, we leverage the concept of traveling wave dynamics within a neural lattice to formulate a theoretical model of neural working memory, study its properties, and its real world implications in AI. The proposed model diverges from traditional approaches, which assume information storage in static, register-like locations updated by interference. Instead, the model stores data as waves that is updated by the wave's boundary conditions. We rigorously examine the model's capabilities in representing and learning state histories, which are vital for learning history-dependent dynamical systems. The findings reveal that the model reliably stores external information and enhances the learning process by addressing the diminishing gradient problem. To understand the model's real-world applicability, we explore two cases: linear boundary condition (LBC) and non-linear, self-attention-driven boundary condition (SBC). The model with the linear boundary condition results in a shift matrix plus low-rank matrix currently used in H3 state space RNN. Further, our experiments with LBC reveal that this matrix is effectively learned by Recurrent Neural Networks (RNNs) through backpropagation when modeling history-dependent dynamical systems. Conversely, the SBC parallels the autoregressive loop of an attention-only transformer with the context vector representing the wave substrate. Collectively, our findings suggest the broader relevance of traveling waves in AI and its potential in advancing neural network architectures.