Generalized Tensor Models for Recurrent Neural Networks

TL;DR

This paper extends the theoretical understanding of Recurrent Neural Networks, demonstrating their universality and depth efficiency with various nonlinearities like ReLU, supported by extensive experiments.

Contribution

It introduces a generalized tensor framework for RNNs, broadening the theoretical analysis to include common nonlinearities and bridging the gap between theory and practice.

Findings

ReLU-based RNNs exhibit universality and depth efficiency.

Theoretical results are validated through extensive experiments.

Generalized tensor models enhance understanding of RNN capabilities.

Abstract

Recurrent Neural Networks (RNNs) are very successful at solving challenging problems with sequential data. However, this observed efficiency is not yet entirely explained by theory. It is known that a certain class of multiplicative RNNs enjoys the property of depth efficiency --- a shallow network of exponentially large width is necessary to realize the same score function as computed by such an RNN. Such networks, however, are not very often applied to real life tasks. In this work, we attempt to reduce the gap between theory and practice by extending the theoretical analysis to RNNs which employ various nonlinearities, such as Rectified Linear Unit (ReLU), and show that they also benefit from properties of universality and depth efficiency. Our theoretical results are verified by a series of extensive computational experiments.