Message Propagation Through Time: An Algorithm for Sequence Dependency Retention in Time Series Modeling

TL;DR

This paper introduces the Message Propagation Through Time (MPTT) algorithm, which enhances RNN training for time series by effectively capturing long-term dependencies while maintaining efficient training times.

Contribution

The paper presents MPTT, a novel algorithm that uses memory modules and filtering policies to improve RNN training on time series data with temporal dependencies.

Findings

MPTT outperforms seven baseline strategies on climate datasets.

MPTT effectively captures long-term temporal dependencies.

MPTT maintains faster training times compared to stateful solutions.

Abstract

Time series modeling, a crucial area in science, often encounters challenges when training Machine Learning (ML) models like Recurrent Neural Networks (RNNs) using the conventional mini-batch training strategy that assumes independent and identically distributed (IID) samples and initializes RNNs with zero hidden states. The IID assumption ignores temporal dependencies among samples, resulting in poor performance. This paper proposes the Message Propagation Through Time (MPTT) algorithm to effectively incorporate long temporal dependencies while preserving faster training times relative to the stateful solutions. MPTT utilizes two memory modules to asynchronously manage initial hidden states for RNNs, fostering seamless information exchange between samples and allowing diverse mini-batches throughout epochs. MPTT further implements three policies to filter outdated and preserve essential information in the hidden states to generate informative initial hidden states for RNNs, facilitating robust training. Experimental results demonstrate that MPTT outperforms seven strategies on four climate datasets with varying levels of temporal dependencies.