Universality and approximation bounds for echo state networks with random weights

TL;DR

This paper proves that echo state networks with random internal weights and general activation functions are universal approximators of continuous causal time-invariant operators, providing explicit constructions and error bounds.

Contribution

It extends universality results to general activation functions and offers explicit sampling procedures for internal weights, with quantification of approximation errors.

Findings

Universality holds for general activation functions under certain conditions.

Explicit sampling procedures are provided for ReLU activation.

Approximation errors are quantified for regular operators.

Abstract

We study the uniform approximation of echo state networks with randomly generated internal weights. These models, in which only the readout weights are optimized during training, have made empirical success in learning dynamical systems. Recent results showed that echo state networks with ReLU activation are universal. In this paper, we give an alternative construction and prove that the universality holds for general activation functions. Specifically, our main result shows that, under certain condition on the activation function, there exists a sampling procedure for the internal weights so that the echo state network can approximate any continuous casual time-invariant operators with high probability. In particular, for ReLU activation, we give explicit construction for these sampling procedures. We also quantify the approximation error of the constructed ReLU echo state networks for sufficiently regular operators.