Rethinking Attention Mechanism in Time Series Classification

TL;DR

This paper introduces a flexible multi-head linear attention mechanism for time series classification that improves efficiency and performance by reducing complexity and noise influence, validated through extensive experiments.

Contribution

The paper proposes a novel multi-head linear attention method with a masking mechanism for better efficiency and noise reduction in time series classification.

Findings

Achieves comparable accuracy to state-of-the-art methods on 85 datasets.

Significantly reduces computational complexity compared to Transformer models.

Demonstrates improved efficiency with lower floating-point operations and fewer parameters.

Abstract

Attention-based models have been widely used in many areas, such as computer vision and natural language processing. However, relevant applications in time series classification (TSC) have not been explored deeply yet, causing a significant number of TSC algorithms still suffer from general problems of attention mechanism, like quadratic complexity. In this paper, we promote the efficiency and performance of the attention mechanism by proposing our flexible multi-head linear attention (FMLA), which enhances locality awareness by layer-wise interactions with deformable convolutional blocks and online knowledge distillation. What's more, we propose a simple but effective mask mechanism that helps reduce the noise influence in time series and decrease the redundancy of the proposed FMLA by masking some positions of each given series proportionally. To stabilize this mechanism, samples are forwarded through the model with random mask layers several times and their outputs are aggregated to teach the same model with regular mask layers. We conduct extensive experiments on 85 UCR2018 datasets to compare our algorithm with 11 well-known ones and the results show that our algorithm has comparable performance in terms of top-1 accuracy. We also compare our model with three Transformer-based models with respect to the floating-point operations per second and number of parameters and find that our algorithm achieves significantly better efficiency with lower complexity.