Propagating Uncertainty through the tanh Function with Application to Reservoir Computing

TL;DR

This paper introduces a novel probabilistic approach to propagate uncertainty through the tanh activation in neural networks, specifically improving reservoir computing by reducing washout time and enhancing performance.

Contribution

It proposes two new methods for uncertainty propagation through tanh and introduces the Probabilistic Echo State Network (PESN) with better average performance.

Findings

PESN outperforms deterministic ESNs with random initialization.

The methods accurately recover means and variances comparable to Monte-Carlo simulations.

Uncertainty propagation reduces washout time in reservoir computing.

Abstract

Many neural networks use the tanh activation function, however when given a probability distribution as input, the problem of computing the output distribution in neural networks with tanh activation has not yet been addressed. One important example is the initialization of the echo state network in reservoir computing, where random initialization of the reservoir requires time to wash out the initial conditions, thereby wasting precious data and computational resources. Motivated by this problem, we propose a novel solution utilizing a moment based approach to propagate uncertainty through an Echo State Network to reduce the washout time. In this work, we contribute two new methods to propagate uncertainty through the tanh activation function and propose the Probabilistic Echo State Network (PESN), a method that is shown to have better average performance than deterministic Echo State Networks given the random initialization of reservoir states. Additionally we test single and multi-step uncertainty propagation of our method on two regression tasks and show that we are able to recover similar means and variances as computed by Monte-Carlo simulations.