Deep Residual Echo State Networks: exploring residual orthogonal connections in untrained Recurrent Neural Networks

TL;DR

This paper introduces Deep Residual Echo State Networks, a new deep untrained RNN architecture with residual orthogonal connections that enhances memory and long-term temporal processing in reservoir computing.

Contribution

The paper proposes a novel deep untrained RNN architecture with residual orthogonal connections, improving memory capacity and stability in Echo State Networks.

Findings

DeepResESNs outperform traditional shallow ESNs on time series tasks.

Residual orthogonal connections enhance long-term memory in untrained RNNs.

Mathematical analysis provides stability conditions for DeepResESNs.

Abstract

Echo State Networks (ESNs) are a particular type of untrained Recurrent Neural Networks (RNNs) within the Reservoir Computing (RC) framework, popular for their fast and efficient learning. However, traditional ESNs often struggle with long-term information processing. In this paper, we introduce a novel class of deep untrained RNNs based on temporal residual connections, called Deep Residual Echo State Networks (DeepResESNs). We show that leveraging a hierarchy of untrained residual recurrent layers significantly boosts memory capacity and long-term temporal modeling. For the temporal residual connections, we consider different orthogonal configurations, including randomly generated and fixed-structure configurations, and we study their effect on network dynamics. A thorough mathematical analysis outlines necessary and sufficient conditions to ensure stable dynamics within DeepResESN. Our experiments on a variety of time series tasks showcase the advantages of the proposed approach over traditional shallow and deep RC.