Reservoir Computing Universality With Stochastic Inputs

TL;DR

This paper proves the universal approximation capabilities of various reservoir computing systems with stochastic inputs, including popular models like echo state networks, under broad $L^p$ criteria.

Contribution

It establishes the universality of linear reservoir systems with polynomial, neural network, trigonometric, and echo state readouts for stochastic inputs, broadening theoretical understanding.

Findings

Linear reservoir systems with polynomial or neural network readouts are universal.

Trigonometric state-affine systems and echo state networks are also universal.

Universality holds without requiring bounded inputs or fading memory.

Abstract

The universal approximation properties with respect to $L ^p $-type criteria of three important families of reservoir computers with stochastic discrete-time semi-infinite inputs is shown. First, it is proved that linear reservoir systems with either polynomial or neural network readout maps are universal. More importantly, it is proved that the same property holds for two families with linear readouts, namely, trigonometric state-affine systems and echo state networks, which are the most widely used reservoir systems in applications. The linearity in the readouts is a key feature in supervised machine learning applications. It guarantees that these systems can be used in high-dimensional situations and in the presence of large datasets. The $L ^p $ criteria used in this paper allow the formulation of universality results that do not necessarily impose almost sure uniform boundedness in the inputs or the fading memory property in the filter that needs to be approximated.