# Deep Residual Echo State Networks: exploring residual orthogonal connections in untrained Recurrent Neural Networks

## TL;DR

This paper introduces Deep Residual Echo State Networks, a new deep untrained RNN architecture with residual orthogonal connections that enhances memory and long-term temporal processing in reservoir computing.

## Contribution

The paper proposes a novel deep untrained RNN architecture with residual orthogonal connections, improving memory capacity and stability in Echo State Networks.

## Key findings

- DeepResESNs outperform traditional shallow ESNs on time series tasks.
- Residual orthogonal connections enhance long-term memory in untrained RNNs.
- Mathematical analysis provides stability conditions for DeepResESNs.

## Abstract

Echo State Networks (ESNs) are a particular type of untrained Recurrent Neural Networks (RNNs) within the Reservoir Computing (RC) framework, popular for their fast and efficient learning. However, traditional ESNs often struggle with long-term information processing. In this paper, we introduce a novel class of deep untrained RNNs based on temporal residual connections, called Deep Residual Echo State Networks (DeepResESNs). We show that leveraging a hierarchy of untrained residual recurrent layers significantly boosts memory capacity and long-term temporal modeling. For the temporal residual connections, we consider different orthogonal configurations, including randomly generated and fixed-structure configurations, and we study their effect on network dynamics. A thorough mathematical analysis outlines necessary and sufficient conditions to ensure stable dynamics within DeepResESN. Our experiments on a variety of time series tasks showcase the advantages of the proposed approach over traditional shallow and deep RC.

---
Source: https://tomesphere.com/paper/2508.21172