The Coupling Strength Is a Scale Parameter in Threshold Power-Law Reservoirs and Does Not Influence Training Accuracy

TL;DR

This paper shows that in threshold power-law recurrent neural networks, the coupling strength acts as a scale parameter that does not affect training accuracy, contrasting with sigmoidal RNNs where it influences dynamics and performance.

Contribution

It demonstrates that for threshold power-law RNNs, the coupling strength is a scale parameter, making the training accuracy invariant to its value, unlike in sigmoidal RNNs.

Findings

Coupling strength acts as a scale parameter in threshold power-law RNNs.

Training accuracy is invariant to the coupling strength in these networks.

This behavior contrasts with sigmoidal RNNs where coupling strength influences dynamics.

Abstract

In reservoir computing, the coupling strength of the initial untrained recurrent neural network (the reservoir) is an important hyperparameter that can be varied for accurate training. A common heuristic is to set this parameter near the ``edge of chaos", where the untrained reservoir is near the transition to chaotic dynamics, and the chaos can be ``tamed". Here, we investigate how the overall connectivity strength should be varied in threshold power-law recurrent neural networks, where the firing rate is 0 below some threshold of the current and is a power function of the current above this threshold. These networks have been previously shown to exhibit chaotic solutions for very small coupling strengths, which may imply that the chaos cannot be tamed at all. We show that for reservoirs constructed with threshold power-law transfer functions, if the reservoir can be trained for one single positive value of the initial reservoir coupling strength, then there exist networks with identical accuracy for all positive coupling strengths, implying that the chaotic dynamics can always be tamed or never be tamed. This is a direct consequence of the coupling strength of threshold power-law RNNs acting as a scale parameter that does not qualitatively influence the dynamics of the system, but only scales all system solutions in magnitude. This is independent of the power of the transfer function, with the exception of Rectified Linear Unit (ReLU) networks. This is in contrast with conventional RNNs/reservoirs employing sigmoidal firing rates, where the strength of the recurrent coupling in the initial reservoir determines the performance on different tasks during training and also influences the network dynamics explicitly.