A Review of Designs and Applications of Echo State Networks

TL;DR

This paper reviews the design, theoretical foundations, and applications of Echo State Networks, highlighting recent advances, combinations with other models, and future challenges in the field.

Contribution

It categorizes ESN-based methods into basic, deep, and combined models, analyzing their design, theory, and applications, and discusses future research directions.

Findings

Deep ESN models integrate deep learning with ESNs.

Combining ESNs with other models can outperform baselines.

ESNs are practical, simple, and theoretically grounded, but require experience for effective application.

Abstract

Recurrent Neural Networks (RNNs) have demonstrated their outstanding ability in sequence tasks and have achieved state-of-the-art in wide range of applications, such as industrial, medical, economic and linguistic. Echo State Network (ESN) is simple type of RNNs and has emerged in the last decade as an alternative to gradient descent training based RNNs. ESN, with a strong theoretical ground, is practical, conceptually simple, easy to implement. It avoids non-converging and computationally expensive in the gradient descent methods. Since ESN was put forward in 2002, abundant existing works have promoted the progress of ESN, and the recently introduced Deep ESN model opened the way to uniting the merits of deep learning and ESNs. Besides, the combinations of ESNs with other machine learning models have also overperformed baselines in some applications. However, the apparent simplicity of ESNs can sometimes be deceptive and successfully applying ESNs needs some experience. Thus, in this paper, we categorize the ESN-based methods to basic ESNs, DeepESNs and combinations, then analyze them from the perspective of theoretical studies, network designs and specific applications. Finally, we discuss the challenges and opportunities of ESNs by summarizing the open questions and proposing possible future works.