Parallelization of Non-linear State-Space Models: Scaling Up Liquid-Resistance Liquid-Capacitance Networks for Efficient Sequence Modeling

TL;DR

LrcSSM introduces a parallelizable non-linear recurrent model with diagonal Jacobian, enabling efficient long-sequence processing, formal gradient stability, and superior performance over existing models on forecasting tasks.

Contribution

The paper proposes LrcSSM, a non-linear recurrent model with diagonal Jacobian for parallel sequence processing, offering stability guarantees and broad applicability.

Findings

LrcSSM processes long sequences with $ ext{O}(TD)$ complexity and $ ext{O}( ext{log} T)$ depth.

LrcSSM outperforms Transformers, LRU, S5, and Mamba on long-range forecasting tasks.

Diagonal Jacobian structure maintains performance without loss compared to dense Jacobian models.

Abstract

We present LrcSSM, a $\textit{non-linear}$ recurrent model that processes long sequences as fast as today's linear state-space layers. By forcing its Jacobian matrix to be diagonal, the full sequence can be solved in parallel, giving $\mathcal{O}(TD)$ computational work and memory and only $\mathcal{O}(\log T)$ sequential depth, for input-sequence length $T$ and a state dimension $D$. Moreover, LrcSSM offers a formal gradient-stability guarantee that other input-varying systems such as Liquid-S4 and Mamba do not provide. Importantly, the diagonal Jacobian structure of our model results in no performance loss compared to the original model with dense Jacobian, and the approach can be generalized to other non-linear recurrent models, demonstrating broader applicability. On a suite of long-range forecasting tasks, we demonstrate that LrcSSM outperforms Transformers, LRU, S5, and Mamba.