Enhancing Channel-Independent Time Series Forecasting via Cross-Variate Patch Embedding

TL;DR

This paper introduces Cross-Variate Patch Embeddings (CVPE), a lightweight module that enhances channel-independent time series forecasting models by effectively capturing cross-variate dependencies, leading to improved performance.

Contribution

The paper proposes a novel CVPE module that injects cross-variate context into CI models through a simple modification of patch embedding, improving forecasting accuracy.

Findings

CVPE improves Time-LLM performance on seven datasets.

Enhanced models outperform baseline without additional complexity.

Cross-variate dependencies are effectively captured by CVPE.

Abstract

Transformers have recently gained popularity in time series forecasting due to their ability to capture long-term dependencies. However, many existing models focus only on capturing temporal dependencies while omitting intricate relationships between variables. Recent models have tried tackling this by explicitly modeling both cross-time and cross-variate dependencies through a sequential or unified attention mechanism, but they are entirely channel dependent (CD) across all layers, making them potentially susceptible to overfitting. To address this, we propose Cross-Variate Patch Embeddings (CVPE), a lightweight CD module that injects cross-variate context into channel-independent (CI) models by simply modifying the patch embedding process. We achieve this by adding a learnable positional encoding and a lightweight router-attention block to the vanilla patch embedding layer. We then integrate CVPE into Time-LLM, a multimodal CI forecasting model, to demonstrate its effectiveness in capturing cross-variate dependencies and enhance the CI model's performance. Extensive experimental results on seven real-world datasets show that our enhanced Time-LLM outperforms the original baseline model simply by incorporating the CVPE module, with no other changes.