Backpropagation Algorithms and Reservoir Computing in Recurrent Neural Networks for the Forecasting of Complex Spatiotemporal Dynamics

TL;DR

This paper compares Reservoir Computing and Backpropagation through time in RNNs for forecasting complex spatiotemporal systems, highlighting their strengths, limitations, and performance differences on chaotic system benchmarks.

Contribution

It provides a comprehensive comparison of RC and BPTT in RNNs for spatiotemporal forecasting, including stability analysis and Lyapunov spectrum quantification.

Findings

RC outperforms BPTT with full state data in accuracy and speed.

BPTT excels with reduced order data and offers better stability.

Lyapunov spectrum of KS equation computed with BPTT matches RC results.

Abstract

We examine the efficiency of Recurrent Neural Networks in forecasting the spatiotemporal dynamics of high dimensional and reduced order complex systems using Reservoir Computing (RC) and Backpropagation through time (BPTT) for gated network architectures. We highlight advantages and limitations of each method and discuss their implementation for parallel computing architectures. We quantify the relative prediction accuracy of these algorithms for the longterm forecasting of chaotic systems using as benchmarks the Lorenz-96 and the Kuramoto-Sivashinsky (KS) equations. We find that, when the full state dynamics are available for training, RC outperforms BPTT approaches in terms of predictive performance and in capturing of the long-term statistics, while at the same time requiring much less training time. However, in the case of reduced order data, large scale RC models can be unstable and more likely than the BPTT algorithms to diverge. In contrast, RNNs trained via BPTT show superior forecasting abilities and capture well the dynamics of reduced order systems. Furthermore, the present study quantifies for the first time the Lyapunov Spectrum of the KS equation with BPTT, achieving similar accuracy as RC. This study establishes that RNNs are a potent computational framework for the learning and forecasting of complex spatiotemporal systems.