Lag Operator SSMs: A Geometric Framework for Structured State Space Modeling

TL;DR

This paper introduces a geometric, first-principles framework for constructing discrete-time structured state space models using a novel lag operator, simplifying design and providing theoretical insights into sequence modeling.

Contribution

The paper presents a new lag operator-based approach for direct construction of discrete-time SSMs, enhancing flexibility and interpretability over traditional methods.

Findings

Recovers the HiPPO recurrence exactly

Provides a modular design space for SSMs

Validated through numerical simulations

Abstract

Structured State Space Models (SSMs), which are at the heart of the recently popular Mamba architecture, are powerful tools for sequence modeling. However, their theoretical foundation relies on a complex, multi-stage process of continuous-time modeling and subsequent discretization, which can obscure intuition. We introduce a direct, first-principles framework for constructing discrete-time SSMs that is both flexible and modular. Our approach is based on a novel lag operator, which geometrically derives the discrete-time recurrence by measuring how the system's basis functions "slide" and change from one timestep to the next. The resulting state matrices are computed via a single inner product involving this operator, offering a modular design space for creating novel SSMs by flexibly combining different basis functions and time-warping schemes. To validate our approach, we demonstrate that a specific instance exactly recovers the recurrence of the influential HiPPO model. Numerical simulations confirm our derivation, providing new theoretical tools for designing flexible and robust sequence models.