Phasor Memory Networks: Stable Backpropagation Through Time for Scalable Explicit Memory

TL;DR

The paper introduces Phasor Memory Networks, a novel architecture that stabilizes backpropagation through time using unitary phasor dynamics, enabling scalable explicit memory for language modeling.

Contribution

It presents a new memory architecture with phase rotation dynamics that prevents gradient divergence, allowing effective long-term memory in neural networks.

Findings

PMNet achieves near 100% exact retrieval on a synthetic Copy-Paste task.

PMNet matches the zero-shot long-context robustness of a larger model despite fewer parameters.

Structural memory issues are addressed through unitary phase rotations, improving stability.

Abstract

For over a decade, explicit memory architectures like the Neural Turing Machine have remained theoretically appealing yet practically intractable for language modeling due to catastrophic gradient instability during Backpropagation Through Time. In this work, we break this stalemate with \textit{Phasor Memory Network} (PMNet), a novel architecture that structurally resolves memory volatility through \textit{Unitary Phasor Dynamics} and \textit{Hierarchical Learnable Anchors}. Rather than relying on brute-force scaling, we present a mechanistic proof-of-concept in a controlled byte-level setting. By constraining recurrent state updates to phase rotations on a complex unit circle, PMNet preserves gradient norms and inherently prevents divergence without the need for specialized initialization. We empirically demonstrate the active actuation of the memory module through a synthetic Copy-Paste task, where PMNet utilizes an expansive \textit{85-slot hierarchical memory tree} ($=\sum^{4}_{h=1}4^{h-1}$) to achieve near 100\% exact retrieval across temporal distances that completely exceed the local sliding window attention's receptive field. Furthermore, despite being a compact 119M parameter model trained on 18.8B tokens, PMNet matches the zero-shot long-context robustness of a Mamba model that is three times larger. Our ablation studies and gradient analyses confirm that the historical failure of explicit memory was a structural alignment problem, which PMNet effectively overcomes, providing a theoretically grounded foundation for scalable sequence modeling.