Enhancing Multivariate Time Series Classifiers through Self-Attention and Relative Positioning Infusion

TL;DR

This paper introduces two novel attention mechanisms to improve deep learning models for multivariate time series classification, demonstrating up to 3.6% accuracy gains on a standard benchmark.

Contribution

It proposes two new attention blocks, including a relative positioning infusion module, that enhance existing deep learning TSC models with minimal added complexity.

Findings

Attention blocks improve accuracy by up to 3.6%

Proposed TPS block outperforms many state-of-the-art methods

Source code and models are publicly available

Abstract

Time Series Classification (TSC) is an important and challenging task for many visual computing applications. Despite the extensive range of methods developed for TSC, relatively few utilized Deep Neural Networks (DNNs). In this paper, we propose two novel attention blocks (Global Temporal Attention and Temporal Pseudo-Gaussian augmented Self-Attention) that can enhance deep learning-based TSC approaches, even when such approaches are designed and optimized for a specific dataset or task. We validate this claim by evaluating multiple state-of-the-art deep learning-based TSC models on the University of East Anglia (UEA) benchmark, a standardized collection of 30 Multivariate Time Series Classification (MTSC) datasets. We show that adding the proposed attention blocks improves base models' average accuracy by up to 3.6%. Additionally, the proposed TPS block uses a new injection module to include the relative positional information in transformers. As a standalone unit with less computational complexity, it enables TPS to perform better than most of the state-of-the-art DNN-based TSC methods. The source codes for our experimental setups and proposed attention blocks are made publicly available.