State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memory

TL;DR

This paper proves that state-space models with layer-wise nonlinearities are universal approximators for sequence-to-sequence tasks, but still suffer from exponential decaying memory, as shown both theoretically and empirically.

Contribution

It demonstrates that stacking nonlinear activations in state-space models enhances their approximation capacity for complex sequences, a novel theoretical and empirical insight.

Findings

Layer-wise nonlinear activation makes state-space models universal approximators.

State-space models still exhibit exponential decaying memory despite nonlinearities.

Theoretical proofs are supported by numerical experiments.

Abstract

State-space models have gained popularity in sequence modelling due to their simple and efficient network structures. However, the absence of nonlinear activation along the temporal direction limits the model's capacity. In this paper, we prove that stacking state-space models with layer-wise nonlinear activation is sufficient to approximate any continuous sequence-to-sequence relationship. Our findings demonstrate that the addition of layer-wise nonlinear activation enhances the model's capacity to learn complex sequence patterns. Meanwhile, it can be seen both theoretically and empirically that the state-space models do not fundamentally resolve the issue of exponential decaying memory. Theoretical results are justified by numerical verifications.