Integrating Time Series into LLMs via Multi-layer Steerable Embedding Fusion for Enhanced Forecasting

TL;DR

This paper introduces MSEF, a framework that enhances LLMs for time series forecasting by integrating TS information at all layers through steerable embedding fusion, leading to significant performance improvements.

Contribution

MSEF is a novel multi-layer fusion method that allows LLMs to access and adapt to time series data at all depths, overcoming shallow integration limitations.

Findings

Achieved an average of 31.8% reduction in MSE across seven benchmarks.

Enabled effective few-shot learning with layer-specific adaptation.

Demonstrated significant performance improvements over baseline methods.

Abstract

Time series (TS) data are ubiquitous across various application areas, rendering time series forecasting (TSF) a fundamental task. With the astounding advances in large language models (LLMs), a variety of methods have been developed to adapt LLMs for time series forecasting. Despite unlocking the potential of LLMs in comprehending TS data, existing methods are inherently constrained by their shallow integration of TS information, wherein LLMs typically access TS representations at shallow layers, primarily at the input layer. This causes the influence of TS representations to progressively fade in deeper layers and eventually leads to ineffective adaptation between textual embeddings and TS representations. In this paper, we propose the Multi-layer Steerable Embedding Fusion (MSEF), a novel framework that enables LLMs to directly access time series patterns at all depths, thereby mitigating the progressive loss of TS information in deeper layers. Specifically, MSEF leverages off-the-shelf time series foundation models to extract semantically rich embeddings, which are fused with intermediate text representations across LLM layers via layer-specific steering vectors. These steering vectors are designed to continuously optimize the alignment between time series and textual modalities and facilitate a layer-specific adaptation mechanism that ensures efficient few-shot learning capabilities. Experimental results on seven benchmarks demonstrate significant performance improvements by MSEF compared with baselines, with an average reduction of 31.8% in terms of MSE. The code is available at https://github.com/One1sAll/MSEF.