Towards a Calculus of Echo State Networks

TL;DR

This paper provides the first exact analytical characterization of the memory curve in echo state networks, linking system structure and input properties to performance, validated by simulations across various configurations.

Contribution

It introduces a novel analytical framework for computing the entire memory curve of echo state networks, surpassing previous bounds and enabling precise performance predictions.

Findings

Analytical formula for the memory curve of echo state networks

Validation of the framework through numerical simulations

Agreement between theoretical predictions and empirical results

Abstract

Reservoir computing is a recent trend in neural networks which uses the dynamical perturbations on the phase space of a system to compute a desired target function. We present how one can formulate an expectation of system performance in a simple class of reservoir computing called echo state networks. In contrast with previous theoretical frameworks, which only reveal an upper bound on the total memory in the system, we analytically calculate the entire memory curve as a function of the structure of the system and the properties of the input and the target function. We demonstrate the precision of our framework by validating its result for a wide range of system sizes and spectral radii. Our analytical calculation agrees with numerical simulations. To the best of our knowledge this work presents the first exact analytical characterization of the memory curve in echo state networks.