Oscillatory State-Space Models

TL;DR

This paper introduces LinOSS, a stable and universal oscillatory state-space model inspired by neural dynamics, enabling efficient long-range sequence modeling and outperforming existing models on various long-horizon tasks.

Contribution

We propose LinOSS, a novel oscillatory state-space model with stable dynamics and universal approximation capabilities for long sequence learning.

Findings

LinOSS outperforms state-of-the-art models on long-range sequence tasks.

LinOSS achieves nearly 2x speedup over Mamba and LRU on sequences of length 50k.

The model demonstrates superior accuracy in long-horizon forecasting and classification.

Abstract

We propose Linear Oscillatory State-Space models (LinOSS) for efficiently learning on long sequences. Inspired by cortical dynamics of biological neural networks, we base our proposed LinOSS model on a system of forced harmonic oscillators. A stable discretization, integrated over time using fast associative parallel scans, yields the proposed state-space model. We prove that LinOSS produces stable dynamics only requiring nonnegative diagonal state matrix. This is in stark contrast to many previous state-space models relying heavily on restrictive parameterizations. Moreover, we rigorously show that LinOSS is universal, i.e., it can approximate any continuous and causal operator mapping between time-varying functions, to desired accuracy. In addition, we show that an implicit-explicit discretization of LinOSS perfectly conserves the symmetry of time reversibility of the underlying dynamics. Together, these properties enable efficient modeling of long-range interactions, while ensuring stable and accurate long-horizon forecasting. Finally, our empirical results, spanning a wide range of time-series tasks from mid-range to very long-range classification and regression, as well as long-horizon forecasting, demonstrate that our proposed LinOSS model consistently outperforms state-of-the-art sequence models. Notably, LinOSS outperforms Mamba and LRU by nearly 2x on a sequence modeling task with sequences of length 50k.