Numerical Analysis of HiPPO-LegS ODE for Deep State Space Models

TL;DR

This paper provides a rigorous mathematical analysis of the HiPPO-LegS ODE used in deep state space models, proving its well-posedness and convergence of numerical schemes for long-range dependency modeling.

Contribution

It establishes the well-posedness of the singular HiPPO-LegS ODE and proves convergence of numerical discretizations, filling a key theoretical gap in the understanding of these models.

Findings

HiPPO-LegS ODE is well-posed despite singularity

Numerical schemes for HiPPO-LegS converge for Riemann integrable inputs

Provides mathematical foundations for deep state space models using HiPPO-LegS

Abstract

In deep learning, the recently introduced state space models utilize HiPPO (High-order Polynomial Projection Operators) memory units to approximate continuous-time trajectories of input functions using ordinary differential equations (ODEs), and these techniques have shown empirical success in capturing long-range dependencies in long input sequences. However, the mathematical foundations of these ODEs, particularly the singular HiPPO-LegS (Legendre Scaled) ODE, and their corresponding numerical discretizations remain unsettled. In this work, we fill this gap by establishing that HiPPO-LegS ODE is well-posed despite its singularity, albeit without the freedom of arbitrary initial conditions. Further, we establish convergence of the associated numerical discretization schemes for Riemann integrable input functions.