Fusing Large Language Models with Temporal Transformers for Time Series Forecasting

TL;DR

This paper introduces a hybrid Transformer architecture that combines large language models and traditional Transformers to improve time series forecasting by integrating semantic and temporal information.

Contribution

It presents a novel fusion method that leverages LLMs' semantic understanding with Transformers' temporal modeling for enhanced forecasting accuracy.

Findings

Outperforms existing models on benchmark datasets

Effectively captures semantic and temporal patterns

Improves forecasting accuracy significantly

Abstract

Recently, large language models (LLMs) have demonstrated powerful capabilities in performing various tasks and thus are applied by recent studies to time series forecasting (TSF) tasks, which predict future values with the given historical time series. Existing LLM-based approaches transfer knowledge learned from text data to time series prediction using prompting or fine-tuning strategies. However, LLMs are proficient at reasoning over discrete tokens and semantic patterns but are not initially designed to model continuous numerical time series data. The gaps between text and time series data lead LLMs to achieve inferior performance to a vanilla Transformer model that is directly trained on TSF data. However, the vanilla Transformers often struggle to learn high-level semantic patterns. In this paper, we design a novel Transformer-based architecture that complementarily leverages LLMs and vanilla Transformers, so as to integrate the high-level semantic representations learned by LLMs into the temporal information encoded by time series Transformers, where a hybrid representation is obtained by fusing the representations from the LLM and the Transformer. The resulting fused representation contains both historical temporal dynamics and semantic variation patterns, allowing our model to predict more accurate future values. Experiments on benchmark datasets demonstrate the effectiveness of the proposed approach.