Linear RNNs Provably Learn Linear Dynamic Systems

TL;DR

This paper provides the first theoretical proof that linear RNNs can learn any stable linear dynamic system efficiently using gradient descent, highlighting the benefits of recurrent structure in learning dynamics.

Contribution

It establishes the first theoretical guarantee for linear RNNs to learn stable linear systems, with polynomial sample and time complexity independent of input sequence length.

Findings

Linear RNNs can learn stable linear systems with polynomial complexity.

The width of the RNN does not depend on input sequence length.

Recurrent structure aids in learning dynamic systems effectively.

Abstract

We study the learning ability of linear recurrent neural networks with Gradient Descent. We prove the first theoretical guarantee on linear RNNs to learn any stable linear dynamic system using any a large type of loss functions. For an arbitrary stable linear system with a parameter $\rho_C$ related to the transition matrix $C$, we show that despite the non-convexity of the parameter optimization loss if the width of the RNN is large enough (and the required width in hidden layers does not rely on the length of the input sequence), a linear RNN can provably learn any stable linear dynamic system with the sample and time complexity polynomial in $\frac{1}{1-\rho_C}$. Our results provide the first theoretical guarantee to learn a linear RNN and demonstrate how can the recurrent structure help to learn a dynamic system.