Domain Adaptation for Time series Transformers using One-step fine-tuning

TL;DR

This paper proposes a one-step fine-tuning method for time series Transformers that improves prediction accuracy in limited data domains by combining pre-training, data augmentation, and gradual unfreezing, outperforming existing methods.

Contribution

It introduces a novel one-step fine-tuning approach with source data integration and gradual unfreezing to enhance Transformer performance on limited data time series tasks.

Findings

Improves indoor temperature prediction by 4.35%.

Enhances wind power prediction by 11.54%.

Outperforms state-of-the-art baselines on two real-world datasets.

Abstract

The recent breakthrough of Transformers in deep learning has drawn significant attention of the time series community due to their ability to capture long-range dependencies. However, like other deep learning models, Transformers face limitations in time series prediction, including insufficient temporal understanding, generalization challenges, and data shift issues for the domains with limited data. Additionally, addressing the issue of catastrophic forgetting, where models forget previously learned information when exposed to new data, is another critical aspect that requires attention in enhancing the robustness of Transformers for time series tasks. To address these limitations, in this paper, we pre-train the time series Transformer model on a source domain with sufficient data and fine-tune it on the target domain with limited data. We introduce the \emph{One-step fine-tuning} approach, adding some percentage of source domain data to the target domains, providing the model with diverse time series instances. We then fine-tune the pre-trained model using a gradual unfreezing technique. This helps enhance the model's performance in time series prediction for domains with limited data. Extensive experimental results on two real-world datasets show that our approach improves over the state-of-the-art baselines by 4.35% and 11.54% for indoor temperature and wind power prediction, respectively.