Learning from Predictions: Fusing Training and Autoregressive Inference for Long-Term Spatiotemporal Forecasts

TL;DR

This paper introduces BPTT-SA, a novel training algorithm for RNNs that reduces error accumulation in long-term spatiotemporal predictions, especially in complex fluid flow systems.

Contribution

We propose the Scheduled Autoregressive BPTT (BPTT-SA) algorithm, improving long-term forecasting accuracy by addressing exposure bias in RNN training.

Findings

BPTT-SA reduces iterative error propagation in convolutional RNNs.

The method enhances long-term prediction accuracy in high-dimensional fluid flows.

Experimental results demonstrate improved stability and accuracy over traditional training methods.

Abstract

Recurrent Neural Networks (RNNs) have become an integral part of modeling and forecasting frameworks in areas like natural language processing and high-dimensional dynamical systems such as turbulent fluid flows. To improve the accuracy of predictions, RNNs are trained using the Backpropagation Through Time (BPTT) method to minimize prediction loss. During testing, RNNs are often used in autoregressive scenarios where the output of the network is fed back into the input. However, this can lead to the exposure bias effect, as the network was trained to receive ground-truth data instead of its own predictions. This mismatch between training and testing is compounded when the state distributions are different, and the train and test losses are measured. To address this, previous studies have proposed solutions for language processing networks with probabilistic predictions. Building on these advances, we propose the Scheduled Autoregressive BPTT (BPTT-SA) algorithm for predicting complex systems. Our results show that BPTT-SA effectively reduces iterative error propagation in Convolutional RNNs and Convolutional Autoencoder RNNs, and demonstrate its capabilities in long-term prediction of high-dimensional fluid flows.