Emergence of Primacy and Recency Effect in Mamba: A Mechanistic Point of View

TL;DR

This paper investigates how primacy and recency effects emerge in Mamba language models, revealing mechanisms of memory retention and forgetting through behavioral analysis and model perturbations.

Contribution

It uncovers three mechanisms underlying memory effects in Mamba models, including long-term memory channels, delta-modulated recurrence, and semantic regularity influences.

Findings

Long-term memory is supported by specific channels encoding early tokens.

Recency effects are governed by delta-modulated recurrence, which is limited by distractors.

Semantic regularity modulates memory allocation, affecting forgetting of intermediate items.

Abstract

We study memory in state-space language models using primacy and recency effects as behavioral tools to uncover how information is retained and forgotten over time. Applying structured recall tasks to the Mamba architecture, we observe a consistent U-shaped accuracy profile, indicating strong performance at the beginning and end of input sequences. We identify three mechanisms that give rise to this pattern. First, long-term memory is supported by a sparse subset of channels within the model's selective state space block, which persistently encode early input tokens and are causally linked to primacy effects. Second, short-term memory is governed by delta-modulated recurrence: recent inputs receive more weight due to exponential decay, but this recency advantage collapses when distractor items are introduced, revealing a clear limit to memory depth. Third, we find that memory allocation is dynamically modulated by semantic regularity: repeated relations in the input sequence shift the delta gating behavior, increasing the tendency to forget intermediate items. We validate these findings via targeted ablations and input perturbations on two large-scale Mamba-based language models: one with 1.4B and another with 7B parameters.