Phase-Associative Memory: Sequence Modeling in Complex Hilbert Space

TL;DR

This paper introduces Phase-Associative Memory (PAM), a complex-valued sequence model inspired by quantum logic, which shows promising parameter efficiency and potential for scalable language modeling.

Contribution

The work presents a novel complex-valued sequence model, PAM, demonstrating stable training and faster loss reduction compared to real-valued models on language tasks.

Findings

PAM trains stably across 5M to 100M parameters on WikiText-103.

PAM exhibits faster loss reduction and narrower gap with real-valued models as parameters increase.

Complex-valued models like PAM could achieve comparable performance with fewer parameters.

Abstract

Experiments probing natural language processing by both humans and LLMs suggest that the meaning of a semantic expression is indeterminate prior to the act of interpretation rather than being specifiable simply as the sum of its parts (i.e. compositionality). This observer-dependent act dynamically actualizes meaning under genuine contextuality more consistent with quantum logical mechanisms than with classical Boolean approaches that assume separability, motivating an approach to language modeling that utilizes a Hilbert space formalism. In this work, we introduce Phase-Associative Memory (PAM) -- a complex-valued sequence model whose state S_t \in \mathbb{C}^{d \times d} accumulates outer products of complex token embeddings retrieved through the conjugate inner product $\mathrm{Re}\langle K \mid Q\rangle / \sqrt{d}$ -- and evaluate it against a structurally matched real-valued ablation. Both architectures train stably across a 5M--100M parameter sweep on WikiText-103 under identical conditions; PAM sits at higher absolute loss at every measured scale but improves more rapidly with parameter count, with power-law exponents of $-0.15$ vs.\ $-0.12$ in loss and $-0.65$ vs.\ $-0.49$ in perplexity that narrow the gap between the two architectures monotonically. Further investigation of complex-valued sequence modeling at larger scales could reveal that the loss plateau characteristic of real-valued state-of-the-art language models (e.g. transformers) is reachable with PAM-style architectures with an order of magnitude fewer parameters than the current frontier ($\sim$1T), implying that similar capabilities are achievable at sizes runnable on consumer-grade hardware.