Deep Echo State Network (DeepESN): A Brief Survey

TL;DR

This paper provides a comprehensive overview of Deep Echo State Networks (DeepESNs), highlighting their efficiency for temporal data processing and their role in understanding hierarchical recurrent dynamics.

Contribution

It offers a summarized survey of recent developments, analysis, and applications of DeepESNs, emphasizing their significance in deep recurrent neural network research.

Findings

DeepESNs enable efficient deep neural network design for temporal data.

Hierarchical recurrent layers in DeepESNs reveal intrinsic state dynamics.

DeepESNs contribute to understanding the bias of depth in RNN architecture.

Abstract

The study of deep recurrent neural networks (RNNs) and, in particular, of deep Reservoir Computing (RC) is gaining an increasing research attention in the neural networks community. The recently introduced Deep Echo State Network (DeepESN) model opened the way to an extremely efficient approach for designing deep neural networks for temporal data. At the same time, the study of DeepESNs allowed to shed light on the intrinsic properties of state dynamics developed by hierarchical compositions of recurrent layers, i.e. on the bias of depth in RNNs architectural design. In this paper, we summarize the advancements in the development, analysis and applications of DeepESNs.