WaLRUS: Wavelets for Long-range Representation Using SSMs

TL;DR

WaLRUS introduces a novel SSM-based framework utilizing Daubechies wavelets to enhance long-range dependency modeling in sequential data, expanding the diversity of SSMs beyond traditional bases.

Contribution

It presents WaLRUS, a new implementation of SaFARi that constructs SSMs from wavelet frames, enabling more flexible and diverse long-range sequence representations.

Findings

Demonstrates improved long-range dependency modeling.

Expands SSM construction to non-orthogonal wavelet bases.

Provides a flexible framework for sequence modeling.

Abstract

State-Space Models (SSMs) have proven to be powerful tools for modeling long-range dependencies in sequential data. While the recent method known as HiPPO has demonstrated strong performance, and formed the basis for machine learning models S4 and Mamba, it remains limited by its reliance on closed-form solutions for a few specific, well-behaved bases. The SaFARi framework generalized this approach, enabling the construction of SSMs from arbitrary frames, including non-orthogonal and redundant ones, thus allowing an infinite diversity of possible "species" within the SSM family. In this paper, we introduce WaLRUS (Wavelets for Long-range Representation Using SSMs), a new implementation of SaFARi built from Daubechies wavelets.