Echo state networks are universal

TL;DR

This paper proves that echo state networks can universally approximate any fading memory system in discrete time, ensuring their broad applicability in modeling complex input/output systems with finite-dimensional neural networks.

Contribution

It establishes the universality of echo state networks for fading memory filters, providing a theoretical foundation for their use in modeling such systems.

Findings

Echo state networks are universal approximants for fading memory filters.

Any fading memory system can be realized by a finite-dimensional neural network.

The proof relies on topological properties of fading memory and reservoir computing systems.

Abstract

This paper shows that echo state networks are universal uniform approximants in the context of discrete-time fading memory filters with uniformly bounded inputs defined on negative infinite times. This result guarantees that any fading memory input/output system in discrete time can be realized as a simple finite-dimensional neural network-type state-space model with a static linear readout map. This approximation is valid for infinite time intervals. The proof of this statement is based on fundamental results, also presented in this work, about the topological nature of the fading memory property and about reservoir computing systems generated by continuous reservoir maps.