Improving Position Encoding of Transformers for Multivariate Time Series Classification

TL;DR

This paper introduces new position encoding methods for transformers in multivariate time series classification, demonstrating significant accuracy improvements over existing models through extensive experiments.

Contribution

It proposes a novel time-specific absolute position encoding (tAPE) and an efficient relative position encoding (eRPE), integrated into a new ConvTran model for improved multivariate time series classification.

Findings

ConvTran outperforms state-of-the-art models on 32 datasets.

tAPE and eRPE are simple, efficient, and easily integrable.

The methods enhance transformer performance in various downstream tasks.

Abstract

Transformers have demonstrated outstanding performance in many applications of deep learning. When applied to time series data, transformers require effective position encoding to capture the ordering of the time series data. The efficacy of position encoding in time series analysis is not well-studied and remains controversial, e.g., whether it is better to inject absolute position encoding or relative position encoding, or a combination of them. In order to clarify this, we first review existing absolute and relative position encoding methods when applied in time series classification. We then proposed a new absolute position encoding method dedicated to time series data called time Absolute Position Encoding (tAPE). Our new method incorporates the series length and input embedding dimension in absolute position encoding. Additionally, we propose computationally Efficient implementation of Relative Position Encoding (eRPE) to improve generalisability for time series. We then propose a novel multivariate time series classification (MTSC) model combining tAPE/eRPE and convolution-based input encoding named ConvTran to improve the position and data embedding of time series data. The proposed absolute and relative position encoding methods are simple and efficient. They can be easily integrated into transformer blocks and used for downstream tasks such as forecasting, extrinsic regression, and anomaly detection. Extensive experiments on 32 multivariate time-series datasets show that our model is significantly more accurate than state-of-the-art convolution and transformer-based models. Code and models are open-sourced at \url{https://github.com/Navidfoumani/ConvTran}.