Towards a Comprehensive Theory of Reservoir Computing

TL;DR

This paper develops a theoretical framework for predicting and optimizing the performance of various reservoir computing models, particularly echo state networks, and introduces a novel ESN model with a non-training readout that outperforms previous models.

Contribution

The paper applies perceptron theory to predict ESN performance, proposes a training-free ESN model, and analyzes the geometry of readout networks in reservoir computing.

Findings

Empirical results confirm the theory's predictions across 30 ESN variants.

The theory enables optimization of memory capacity in ESNs.

A new ESN model with a non-training readout outperforms earlier models.

Abstract

In reservoir computing, an input sequence is processed by a recurrent neural network, the reservoir, which transforms it into a spatial pattern that a shallow readout network can then exploit for tasks such as memorization and time-series prediction or classification. Echo state networks (ESN) are a model class in which the reservoir is a traditional artificial neural network. This class contains many model types, each with sets of hyperparameters. Selecting models and parameter settings for particular applications requires a theory for predicting and comparing performances. Here, we demonstrate that recent developments of perceptron theory can be used to predict the memory capacity and accuracy of a wide variety of ESN models, including reservoirs with linear neurons, sigmoid nonlinear neurons, different types of recurrent matrices, and different types of readout networks. Across thirty variants of ESNs, we show that empirical results consistently confirm the theory's predictions. As a practical demonstration, the theory is used to optimize memory capacity of an ESN in the entire joint parameter space. Further, guided by the theory, we propose a novel ESN model with a readout network that does not require training, and which outperforms earlier ESN models without training. Finally, we characterize the geometry of the readout networks in ESNs, which reveals that many ESN models exhibit a similar regular simplex geometry as has been observed in the output weights of deep neural networks.