TiVaT: A Transformer with a Single Unified Mechanism for Capturing Asynchronous Dependencies in Multivariate Time Series Forecasting

TL;DR

TiVaT introduces a unified Transformer architecture with a Joint-Axis attention module that simultaneously models temporal and inter-variate dependencies, effectively capturing asynchronous interactions in multivariate time series forecasting.

Contribution

The paper presents TiVaT, a novel Transformer with a single module that models asynchronous dependencies jointly, improving over channel-dependent models.

Findings

TiVaT outperforms existing models on multiple datasets.

The Joint-Axis attention effectively captures asynchronous dependencies.

Distance-aware sampling enhances pattern extraction and noise reduction.

Abstract

Multivariate time series (MTS) forecasting is vital across various domains but remains challenging due to the need to simultaneously model temporal and inter-variate dependencies. Existing channel-dependent models, where Transformer-based models dominate, process these dependencies separately, limiting their capacity to capture complex interactions such as lead-lag dynamics. To address this issue, we propose TiVaT (Time-variate Transformer), a novel architecture incorporating a single unified module, a Joint-Axis (JA) attention module, that concurrently processes temporal and variate modeling. The JA attention module dynamically selects relevant features to particularly capture asynchronous interactions. In addition, we introduce distance-aware time-variate sampling in the JA attention, a novel mechanism that extracts significant patterns through a learned 2D embedding space while reducing noise. Extensive experiments demonstrate TiVaT's overall performance across diverse datasets, particularly excelling in scenarios with intricate asynchronous dependencies.