Time-Warping Recurrent Neural Networks for Transfer Learning

TL;DR

This paper introduces a novel transfer learning method for RNNs using time-warping, enabling models to adapt to different time scales in physical systems while maintaining accuracy.

Contribution

It proposes a new time-warping transfer learning approach for RNNs, supported by theoretical proof and practical evaluation on wildfire fuel moisture prediction.

Findings

Time-warping transfer learning achieves accuracy comparable to existing methods.

The method modifies only a small fraction of parameters during transfer.

The approach is effective across a wide range of time scales in practical applications.

Abstract

Dynamical systems describe how a physical system evolves over time. Physical processes can evolve faster or slower in different environmental conditions. We use time-warping as rescaling the time in a model of a physical system. This thesis proposes a new method of transfer learning for Recurrent Neural Networks (RNNs) based on time-warping. We prove that for a class of linear, first-order differential equations known as time lag models, an LSTM can approximate these systems with any desired accuracy, and the model can be time-warped while maintaining the approximation accuracy. The Time-Warping method of transfer learning is then evaluated in an applied problem on predicting fuel moisture content (FMC), an important concept in wildfire modeling. An RNN with LSTM recurrent layers is pretrained on fuels with a characteristic time scale of 10 hours, where there are large quantities of data available for training. The RNN is then modified with transfer learning to generate predictions for fuels with characteristic time scales of 1 hour, 100 hours, and 1000 hours. The Time-Warping method is evaluated against several known methods of transfer learning. The Time-Warping method produces predictions with an accuracy level comparable to the established methods, despite modifying only a small fraction of the parameters that the other methods modify.