Orthogonal Echo State Networks and stochastic evaluations of likelihoods

TL;DR

This paper explores probabilistic likelihood estimation in echo state networks with orthogonal recurrent connectivity, analyzing how input strength, recurrent activity, and connectivity types affect inference quality and network performance.

Contribution

It introduces a measure based on mutual information and evaluates different recurrent connectivity types, highlighting the advantages of orthogonal matrices for inference.

Findings

Orthogonal recurrent connectivity yields the best inference performance.

Network sensitivity depends on input strength and recurrent activity balance.

Orthogonal matrices scale well with network size.

Abstract

We report about probabilistic likelihood estimates that are performed on time series using an echo state network with orthogonal recurrent connectivity. The results from tests using synthetic stochastic input time series with temporal inference indicate that the capability of the network to infer depends on the balance between input strength and recurrent activity. This balance has an influence on the network with regard to the quality of inference from the short term input history versus inference that accounts for influences that date back a long time. Sensitivity of such networks against noise and the finite accuracy of network states in the recurrent layer are investigated. In addition, a measure based on mutual information between the output time series and the reservoir is introduced. Finally, different types of recurrent connectivity are evaluated. Orthogonal matrices show the best results of all investigated connectivity types overall, but also in the way how the network performance scales with the size of the recurrent layer.