Analysis of Memory Capacity for Deep Echo State Networks

TL;DR

This paper analyzes the memory capacity of new deep ESN architectures, showing how parallel and series configurations affect memory and prediction accuracy, with parallel ESNs reducing error more significantly.

Contribution

It introduces and evaluates two novel deep ESN architectures, parallel and series, and analyzes their memory capacities and prediction performance.

Findings

Parallel ESNs have similar memory capacity to shallow ESNs.

Series ESNs have smaller memory capacity than shallow ESNs.

Parallel ESNs reduce normalized root mean square error by 38.5%.

Abstract

In this paper, the echo state network (ESN) memory capacity, which represents the amount of input data an ESN can store, is analyzed for a new type of deep ESNs. In particular, two deep ESN architectures are studied. First, a parallel deep ESN is proposed in which multiple reservoirs are connected in parallel allowing them to average outputs of multiple ESNs, thus decreasing the prediction error. Then, a series architecture ESN is proposed in which ESN reservoirs are placed in cascade that the output of each ESN is the input of the next ESN in the series. This series ESN architecture can capture more features between the input sequence and the output sequence thus improving the overall prediction accuracy. Fundamental analysis shows that the memory capacity of parallel ESNs is equivalent to that of a traditional shallow ESN, while the memory capacity of series ESNs is smaller than that of a traditional shallow ESN.In terms of normalized root mean square error, simulation results show that the parallel deep ESN achieves 38.5% reduction compared to the traditional shallow ESN while the series deep ESN achieves 16.8% reduction.