CALF: Aligning LLMs for Time Series Forecasting via Cross-modal Fine-Tuning

TL;DR

CALF introduces a novel cross-modal fine-tuning framework for LLM-based multivariate time series forecasting, effectively reducing distribution discrepancies between textual and temporal data, leading to state-of-the-art results with low complexity.

Contribution

The paper proposes a cross-modal alignment method for LLMs in time series forecasting, addressing distribution gaps between text and time series inputs for improved performance.

Findings

Achieves state-of-the-art forecasting accuracy.

Demonstrates strong few-shot and zero-shot capabilities.

Maintains low computational complexity.

Abstract

Deep learning (e.g., Transformer) has been widely and successfully used in multivariate time series forecasting (MTSF). Unlike existing methods that focus on training models from a single modal of time series input, large language models (LLMs) based MTSF methods with cross-modal text and time series input have recently shown great superiority, especially with limited temporal data. However, current LLM-based MTSF methods usually focus on adapting and fine-tuning LLMs, while neglecting the distribution discrepancy between textual and temporal input tokens, thus leading to sub-optimal performance. To address this issue, we propose a novel Cross-Modal LLM Fine-Tuning (CALF) framework for MTSF by reducing the distribution discrepancy between textual and temporal data, which mainly consists of the temporal target branch with temporal input and the textual source branch with aligned textual input. To reduce the distribution discrepancy, we develop the cross-modal match module to first align cross-modal input distributions. Additionally, to minimize the modality distribution gap in both feature and output spaces, feature regularization loss is developed to align the intermediate features between the two branches for better weight updates, while output consistency loss is introduced to allow the output representations of both branches to correspond effectively. Thanks to the modality alignment, CALF establishes state-of-the-art performance for both long-term and short-term forecasting tasks with low computational complexity, and exhibiting favorable few-shot and zero-shot abilities similar to that in LLMs. Code is available at https://github.com/Hank0626/LLaTA.