IRNN: Innovation-driven Recurrent Neural Network for Time-Series Data Modeling and Prediction

TL;DR

IRNN introduces an innovation-driven approach to RNNs for time-series prediction, utilizing past prediction errors as additional inputs, which significantly enhances accuracy without added training complexity.

Contribution

The paper proposes IRNN, a novel RNN architecture that incorporates innovations from Kalman filters, along with a new training algorithm IU-BPTT, improving time-series prediction performance.

Findings

IRNN outperforms traditional RNNs on benchmark datasets.

Incorporating innovations improves prediction accuracy.

Training cost remains comparable to standard RNN training.

Abstract

Many real-world datasets are time series that are sequentially collected and contain rich temporal information. Thus, a common interest in practice is to capture dynamics of time series and predict their future evolutions. To this end, the recurrent neural network (RNN) has been a prevalent and effective machine learning option, which admits a nonlinear state-space model representation. Motivated by the resemblance between RNN and Kalman filter (KF) for linear state-space models, we propose in this paper Innovation-driven RNN (IRNN), a novel RNN architecture tailored to time-series data modeling and prediction tasks. By adapting the concept of "innovation" from KF to RNN, past prediction errors are adopted as additional input signals to update hidden states of RNN and boost prediction performance. Since innovation data depend on network parameters, existing training algorithms for RNN do not apply to IRNN straightforwardly. Thus, a tailored training algorithm dubbed input updating-based back-propagation through time (IU-BPTT) is further proposed, which alternates between updating innovations and optimizing network parameters via gradient descent. Experiments on real-world benchmark datasets show that the integration of innovations into various forms of RNN leads to remarkably improved prediction accuracy of IRNN without increasing the training cost substantially.