QKCV Attention: Enhancing Time Series Forecasting with Static Categorical Embeddings for Both Lightweight and Pre-trained Foundation Models

TL;DR

This paper introduces QKCV attention, a novel module that incorporates static categorical embeddings into time series models, improving forecasting accuracy and enabling efficient fine-tuning of pre-trained models with minimal updates.

Contribution

The paper proposes QKCV attention, a versatile plug-in that enhances existing models with category-specific information and facilitates lightweight fine-tuning of foundation models.

Findings

QKCV improves forecasting accuracy across multiple datasets.

QKCV enables effective fine-tuning by updating only static embeddings.

QKCV is compatible with various attention-based models.

Abstract

In real-world time series forecasting tasks, category information plays a pivotal role in capturing inherent data patterns. This paper introduces QKCV (Query-Key-Category-Value) attention, an extension of the traditional QKV framework that incorporates a static categorical embedding C to emphasize category-specific information. As a versatile plug-in module, QKCV enhances the forecasting accuracy of attention-based models (e.g., Vanilla Transformer, Informer, PatchTST, TFT) across diverse real-world datasets. Furthermore, QKCV demonstrates remarkable adaptability in fine-tuning univariate time series foundation model by solely updating the static embedding C while preserving pretrained weights, thereby reducing computational overhead and achieving superior fine-tuning performance.