Comparing Prior and Learned Time Representations in Transformer Models of Timeseries

TL;DR

This paper compares fixed and learned time representations in Transformer models for time series prediction, highlighting challenges in encoding prior knowledge and emphasizing the need for human-in-the-loop approaches.

Contribution

It introduces a comparison between fixed and learned time representations in Transformers for time series, revealing difficulties in encoding prior knowledge and suggesting future research directions.

Findings

Fixed representations perform well on known periodicities.

Learned representations face challenges in encoding prior knowledge.

Human-in-the-loop methods may enhance robustness.

Abstract

What sets timeseries analysis apart from other machine learning exercises is that time representation becomes a primary aspect of the experiment setup, as it must adequately represent the temporal relations that are relevant for the application at hand. In the work described here we study wo different variations of the Transformer architecture: one where we use the fixed time representation proposed in the literature and one where the time representation is learned from the data. Our experiments use data from predicting the energy output of solar panels, a task that exhibits known periodicities (daily and seasonal) that is straight-forward to encode in the fixed time representation. Our results indicate that even in an experiment where the phenomenon is well-understood, it is difficult to encode prior knowledge due to side-effects that are difficult to mitigate. We conclude that research work is needed to work the human into the learning loop in ways that improve the robustness and trust-worthiness of the network.