Dataset-Driven Channel Masks in Transformers for Multivariate Time Series

TL;DR

This paper introduces dataset-specific channel masks in Transformer models to better capture channel dependencies in multivariate time series, improving modeling accuracy across diverse datasets.

Contribution

It proposes a novel dataset-driven approach using channel masks that refine attention mechanisms in Transformers for multivariate time series analysis.

Findings

Channel masks improve dependency modeling in multivariate TS.

The approach enhances Transformer performance across multiple datasets.

Code implementation is publicly available at the provided GitHub repository.

Abstract

Recent advancements in foundation models have been successfully extended to the time series (TS) domain, facilitated by the emergence of large-scale TS datasets. However, previous efforts have primarily Capturing channel dependency (CD) is essential for modeling multivariate time series (TS), and attention-based methods have been widely employed for this purpose. Nonetheless, these methods primarily focus on modifying the architecture, often neglecting the importance of dataset-specific characteristics. In this work, we introduce the concept of partial channel dependence (PCD) to enhance CD modeling in Transformer-based models by leveraging dataset-specific information to refine the CD captured by the model. To achieve PCD, we propose channel masks (CMs), which are integrated into the attention matrices of Transformers via element-wise multiplication. CMs consist of two components: 1) a similarity matrix that captures relationships between the channels, and 2) dataset-specific and learnable domain parameters that refine the similarity matrix. We validate the effectiveness of PCD across diverse tasks and datasets with various backbones. Code is available at this repository: https://github.com/YonseiML/pcd.