Memory Capacity of Nonlinear Recurrent Networks: Is it Informative?

TL;DR

This paper investigates the memory capacity of nonlinear recurrent neural networks, revealing that it varies arbitrarily and questions the usefulness of existing metrics for assessing their performance in processing stochastic signals.

Contribution

It demonstrates that the memory capacity of nonlinear RNNs depends solely on input scale, challenging the relevance of traditional memory capacity metrics.

Findings

Memory capacity varies arbitrarily within bounds

Existing metrics lack practical value for nonlinear RNNs

Memory capacity depends only on input scale

Abstract

The total memory capacity (MC) of linear recurrent neural networks (RNNs) has been proven to be equal to the rank of the corresponding Kalman controllability matrix, and it is almost surely maximal for connectivity and input weight matrices drawn from regular distributions. This fact questions the usefulness of this metric in distinguishing the performance of linear RNNs in the processing of stochastic signals. This work shows that the MC of random nonlinear RNNs yields arbitrary values within established upper and lower bounds depending exclusively on the scale of the input process. This confirms that the existing definition of MC in linear and nonlinear cases has no practical value.