Emergence of the Primacy Effect in Structured State-Space Models

TL;DR

This paper discovers that structured state-space models (SSMs) tend to remember initial inputs more strongly than recent ones, contradicting their theoretical design, which has implications for understanding their memory mechanisms.

Contribution

The study reveals the emergence of a primacy effect in SSMs, challenging existing theories about their memory decay and persistence.

Findings

SSMs exhibit a primacy effect, favoring initial inputs over recent ones.

This bias appears despite the theoretical expectation of monotonically decaying memory.

The primacy effect was observed on a synthetic memorization task.

Abstract

Structured state-space models (SSMs) have been developed to offer more persistent memory retention than traditional recurrent neural networks, while maintaining real-time inference capabilities and addressing the time-complexity limitations of Transformers. Despite this intended persistence, the memory mechanism of canonical SSMs is theoretically designed to decay monotonically over time, meaning that more recent inputs are expected to be retained more accurately than earlier ones. Contrary to this theoretical expectation, however, the present study reveals a counterintuitive finding: when trained and evaluated on a synthetic, statistically balanced memorization task, SSMs predominantly preserve the *initially* presented data in memory. This pattern of memory bias, known as the *primacy effect* in psychology, presents a non-trivial challenge to the current theoretical understanding of SSMs and opens new avenues for future research.