On the Role of Depth in the Expressivity of RNNs

TL;DR

This paper investigates how depth enhances the expressivity and memory capacity of RNNs and 2RNNs, showing that depth enables more complex transformations and better information retention.

Contribution

It provides a formal analysis of depth's role in RNNs and 2RNNs, demonstrating increased expressivity and memory capacity, and highlights the unique benefits of multiplicative interactions.

Findings

Depth increases RNNs' memory capacity efficiently.

2RNNs perform polynomial transformations with degree growing with depth.

Multiplicative interactions cannot be replaced by layerwise nonlinearities.

Abstract

The benefits of depth in feedforward neural networks are well known: composing multiple layers of linear transformations with nonlinear activations enables complex computations. While similar effects are expected in recurrent neural networks (RNNs), it remains unclear how depth interacts with recurrence to shape expressive power. Here, we formally show that depth increases RNNs' memory capacity efficiently with respect to the number of parameters, thus enhancing expressivity both by enabling more complex input transformations and improving the retention of past information. We broaden our analysis to 2RNNs, a generalization of RNNs with multiplicative interactions between inputs and hidden states. Unlike RNNs, which remain linear without nonlinear activations, 2RNNs perform polynomial transformations whose maximal degree grows with depth. We further show that multiplicative interactions cannot, in general, be replaced by layerwise nonlinearities. Finally, we validate these insights empirically on synthetic and real-world tasks.