Continuous-Time Linear Positional Embedding for Irregular Time Series Forecasting

TL;DR

This paper introduces CTLPE, a novel continuous-time linear positional embedding method designed for irregular time series forecasting, effectively capturing temporal information and outperforming existing techniques.

Contribution

The paper proposes a continuous linear positional embedding approach for transformers to handle irregular time series, with empirical and theoretical advantages over previous methods.

Findings

CTLPE outperforms existing methods on various datasets.

Linear continuous functions are empirically superior for positional encoding.

Theoretical properties support the effectiveness of CTLPE.

Abstract

Irregularly sampled time series forecasting, characterized by non-uniform intervals, is prevalent in practical applications. However, previous research have been focused on regular time series forecasting, typically relying on transformer architectures. To extend transformers to handle irregular time series, we tackle the positional embedding which represents the temporal information of the data. We propose CTLPE, a method learning a continuous linear function for encoding temporal information. The two challenges of irregular time series, inconsistent observation patterns and irregular time gaps, are solved by learning a continuous-time function and concise representation of position. Additionally, the linear continuous function is empirically shown superior to other continuous functions by learning a neural controlled differential equation-based positional embedding, and theoretically supported with properties of ideal positional embedding. CTLPE outperforms existing techniques across various irregularly-sampled time series datasets, showcasing its enhanced efficacy.