HiPPO Zoo: Explicit Memory Mechanisms for Interpretable State Space Models

TL;DR

This paper revisits the HiPPO framework to develop explicit, interpretable polynomial memory models that enhance sequence modeling by supporting adaptive and associative memory, bridging the gap between interpretability and modern SSM capabilities.

Contribution

The paper introduces a unified framework called 'HiPPO zoo' with five extensions that make memory mechanisms explicit and interpretable in polynomial-based sequence models.

Findings

Models support online adaptive memory allocation.

Models achieve efficient streaming training.

Capabilities of modern SSMs are realized through explicit polynomial structures.

Abstract

Representing the past in a compressed, efficient, and informative manner is a central problem for systems trained on sequential data. The HiPPO framework, originally proposed by Gu & Dao et al., provides a principled approach to sequential compression by projecting signals onto orthogonal polynomial (OP) bases via structured linear ordinary differential equations. Subsequent works have embedded these dynamics in state space models (SSMs), where HiPPO structure serves as an initialization. Nonlinear successors of these SSM methods such as Mamba are state-of-the-art for many tasks with long-range dependencies, but the mechanisms by which they represent and prioritize history remain largely implicit. In this work, we revisit the HiPPO framework with the goal of making these mechanisms explicit. We show how polynomial representations of history can be extended to support capabilities of modern SSMs such as adaptive allocation of memory and associative memory while retaining direct interpretability in the OP basis. We introduce a unified framework comprising five such extensions, which we collectively refer to as a "HiPPO zoo." Each extension exposes a specific modeling capability through an explicit, interpretable modification of the HiPPO framework. The resulting models adapt their memory online and train in streaming settings with efficient updates. We illustrate the behaviors and modeling advantages of these extensions through a range of synthetic sequence modeling tasks, demonstrating that capabilities typically associated with modern SSMs can be realized through explicit, interpretable polynomial memory structures.