Stable Recurrent Models

TL;DR

This paper investigates stable recurrent neural networks, proving their approximation by feed-forward models and demonstrating comparable performance on sequence tasks, thus highlighting the importance of stability in sequence learning.

Contribution

It provides a theoretical proof that stable RNNs are well approximated by feed-forward networks and empirically shows their competitive performance on benchmark tasks.

Findings

Stable RNNs are well approximated by feed-forward networks.

Stable RNNs perform comparably to unstable ones on sequence tasks.

Stability explains the success of replacing RNNs with feed-forward models.

Abstract

Stability is a fundamental property of dynamical systems, yet to this date it has had little bearing on the practice of recurrent neural networks. In this work, we conduct a thorough investigation of stable recurrent models. Theoretically, we prove stable recurrent neural networks are well approximated by feed-forward networks for the purpose of both inference and training by gradient descent. Empirically, we demonstrate stable recurrent models often perform as well as their unstable counterparts on benchmark sequence tasks. Taken together, these findings shed light on the effective power of recurrent networks and suggest much of sequence learning happens, or can be made to happen, in the stable regime. Moreover, our results help to explain why in many cases practitioners succeed in replacing recurrent models by feed-forward models.