Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers

TL;DR

This paper introduces the Linear State-Space Layer (LSSL), a simple yet powerful sequence model that unifies and extends recurrent, convolutional, and neural differential equation approaches, achieving state-of-the-art results on various long-sequence tasks.

Contribution

The paper proposes LSSL, a novel continuous-time model that generalizes existing sequence models and incorporates long-range memory, demonstrating superior performance on multiple benchmarks.

Findings

LSSL achieves state-of-the-art results on long-sequence benchmarks.

LSSL outperforms prior methods on speech classification with 16000-length sequences.

LSSL surpasses baselines with handcrafted features on shorter sequences.

Abstract

Recurrent neural networks (RNNs), temporal convolutions, and neural differential equations (NDEs) are popular families of deep learning models for time-series data, each with unique strengths and tradeoffs in modeling power and computational efficiency. We introduce a simple sequence model inspired by control systems that generalizes these approaches while addressing their shortcomings. The Linear State-Space Layer (LSSL) maps a sequence $u \mapsto y$ by simply simulating a linear continuous-time state-space representation $\dot{x} = Ax + Bu, y = Cx + Du$. Theoretically, we show that LSSL models are closely related to the three aforementioned families of models and inherit their strengths. For example, they generalize convolutions to continuous-time, explain common RNN heuristics, and share features of NDEs such as time-scale adaptation. We then incorporate and generalize recent theory on continuous-time memorization to introduce a trainable subset of structured matrices $A$ that endow LSSLs with long-range memory. Empirically, stacking LSSL layers into a simple deep neural network obtains state-of-the-art results across time series benchmarks for long dependencies in sequential image classification, real-world healthcare regression tasks, and speech. On a difficult speech classification task with length-16000 sequences, LSSL outperforms prior approaches by 24 accuracy points, and even outperforms baselines that use hand-crafted features on 100x shorter sequences.