TiMi: Empower Time Series Transformers with Multimodal Mixture of Experts

TL;DR

TiMi enhances multimodal time series forecasting by integrating textual causal information through LLM-guided inferences and a Multimodal Mixture-of-Experts module, achieving state-of-the-art results across multiple benchmarks.

Contribution

The paper introduces TiMi, a novel framework combining LLM-based causal reasoning with a lightweight MMoE module for effective multimodal forecasting without explicit modality alignment.

Findings

Achieves state-of-the-art performance on 16 real-world benchmarks.

Effectively incorporates textual causal information into time series predictions.

Demonstrates strong adaptability and interpretability in multimodal forecasting.

Abstract

Multimodal time series forecasting has garnered significant attention for its potential to provide more accurate predictions than traditional single-modality models by leveraging rich information inherent in other modalities. However, due to fundamental challenges in modality alignment, existing methods often struggle to effectively incorporate multimodal data into predictions, particularly textual information that has a causal influence on time series fluctuations, such as emergency reports and policy announcements. In this paper, we reflect on the role of textual information in numerical forecasting and propose Time series transformers with Multimodal Mixture-of-Experts, TiMi, to unleash the causal reasoning capabilities of LLMs. Concretely, TiMi utilizes LLMs to generate inferences on future developments, which serve as guidance for time series forecasting. To seamlessly integrate both exogenous factors and time series into predictions, we introduce a Multimodal Mixture-of-Experts (MMoE) module as a lightweight plug-in to empower Transformer-based time series models for multimodal forecasting, eliminating the need for explicit representation-level alignment. Experimentally, our proposed TiMi demonstrates consistent state-of-the-art performance on sixteen real-world multimodal forecasting benchmarks, outperforming advanced baselines while offering both strong adaptability and interpretability.