Investigation of Proper Orthogonal Decomposition for Echo State Networks

TL;DR

This paper investigates the use of Proper Orthogonal Decomposition (POD) to reduce the complexity of Echo State Networks (ESN), demonstrating that POD can maintain performance while significantly speeding up computations.

Contribution

It introduces and evaluates a POD-based model order reduction method for ESNs, showing improved efficiency with minimal performance loss.

Findings

POD-reduced ESNs have similar or better memory capacity than full ESNs.

POD reduction achieves approximately 80% speedup in computation.

Performance loss in POD-reduced ESNs is minimal compared to original ESNs.

Abstract

Echo State Networks (ESN) are a type of Recurrent Neural Network that yields promising results in representing time series and nonlinear dynamic systems. Although they are equipped with a very efficient training procedure, Reservoir Computing strategies, such as the ESN, require high-order networks, i.e., many neurons, resulting in a large number of states that are magnitudes higher than the number of model inputs and outputs. A large number of states not only makes the time-step computation more costly but also may pose robustness issues, especially when applying ESNs to problems such as Model Predictive Control (MPC) and other optimal control problems. One way to circumvent this complexity issue is through Model Order Reduction strategies such as the Proper Orthogonal Decomposition (POD) and its variants (POD-DEIM), whereby we find an equivalent lower order representation to an already trained high dimension ESN. To this end, this work aims to investigate and analyze the performance of POD methods in Echo State Networks, evaluating their effectiveness through the Memory Capacity (MC) of the POD-reduced network compared to the original (full-order) ESN. We also perform experiments on two numerical case studies: a NARMA10 difference equation and an oil platform containing two wells and one riser. The results show that there is little loss of performance comparing the original ESN to a POD-reduced counterpart and that the performance of a POD-reduced ESN tends to be superior to a normal ESN of the same size. Also, the POD-reduced network achieves speedups of around $80\%$ compared to the original ESN.