Spectral Text Fusion: A Frequency-Aware Approach to Multimodal Time-Series Forecasting

TL;DR

This paper introduces SpecTF, a frequency-aware framework for multimodal time-series forecasting that effectively combines textual and numerical data by spectral decomposition and cross-attention, improving accuracy with fewer parameters.

Contribution

The paper presents a novel spectral decomposition-based method that integrates textual context into time series forecasting in the frequency domain, addressing multiscale temporal influences.

Findings

Outperforms state-of-the-art models on multiple datasets.

Uses fewer parameters than existing methods.

Effectively captures both short-term and long-term patterns.

Abstract

Multimodal time series forecasting is crucial in real-world applications, where decisions depend on both numerical data and contextual signals. The core challenge is to effectively combine temporal numerical patterns with the context embedded in other modalities, such as text. While most existing methods align textual features with time-series patterns one step at a time, they neglect the multiscale temporal influences of contextual information such as time-series cycles and dynamic shifts. This mismatch between local alignment and global textual context can be addressed by spectral decomposition, which separates time series into frequency components capturing both short-term changes and long-term trends. In this paper, we propose SpecTF, a simple yet effective framework that integrates the effect of textual data on time series in the frequency domain. Our method extracts textual embeddings, projects them into the frequency domain, and fuses them with the time series' spectral components using a lightweight cross-attention mechanism. This adaptively reweights frequency bands based on textual relevance before mapping the results back to the temporal domain for predictions. Experimental results demonstrate that SpecTF significantly outperforms state-of-the-art models across diverse multi-modal time series datasets while utilizing considerably fewer parameters. Code is available at https://github.com/hiepnh137/SpecTF.