SMR: State Memory Replay for Long Sequence Modeling

TL;DR

This paper introduces SMR, a novel memory replay mechanism that enhances state space models' ability to handle non-uniform sampling in long sequence modeling, improving stability and generalization.

Contribution

The paper proposes SMR, a plug-and-play memory mechanism that addresses NSS issues in SSMs, enabling stable modeling of varying sampling points in long sequences.

Findings

SMR improves long-range modeling performance.

SMR enhances stability across different sampling points.

Experimental results show consistent gains in language modeling and LRA tasks.

Abstract

Despite the promising performance of state space models (SSMs) in long sequence modeling, limitations still exist. Advanced SSMs like S5 and S6 (Mamba) in addressing non-uniform sampling, their recursive structures impede efficient SSM computation via convolution. To overcome compatibility limitations in parallel convolutional computation, this paper proposes a novel non-recursive non-uniform sample processing strategy. Theoretical analysis of SSMs through the lens of Event-Triggered Control (ETC) theory reveals the Non-Stable State (NSS) problem, where deviations from sampling point requirements lead to error transmission and accumulation, causing the divergence of the SSM's hidden state. Our analysis further reveals that adjustments of input sequences with early memories can mitigate the NSS problem, achieving Sampling Step Adaptation (SSA). Building on this insight, we introduce a simple yet effective plug-and-play mechanism, State Memory Replay (SMR), which utilizes learnable memories to adjust the current state with multi-step information for generalization at sampling points different from those in the training data. This enables SSMs to stably model varying sampling points. Experiments on long-range modeling tasks in autoregressive language modeling and Long Range Arena demonstrate the general effectiveness of the SMR mechanism for a series of SSM models.