When Does Multimodality Lead to Better Time Series Forecasting?

TL;DR

This study systematically evaluates when incorporating textual data improves time series forecasting, revealing that benefits depend on model capacity, data characteristics, and alignment strategies, and are not universally guaranteed.

Contribution

The paper provides a comprehensive analysis of conditions under which multimodal (text and time series) models improve forecasting, offering data-agnostic insights across diverse domains.

Findings

Multimodal benefits depend on model capacity and data quality.

Alignment strategies influence the effectiveness of multimodal models.

Gains are more likely with sufficient data and complementary text signals.

Abstract

Recently, there has been growing interest in incorporating textual information into foundation models for time series forecasting. However, it remains unclear whether and under what conditions such multimodal integration consistently yields gains. We systematically investigate these questions across a diverse benchmark of 16 forecasting tasks spanning 7 domains, including health, environment, and economics. We evaluate two popular multimodal forecasting paradigms: aligning-based methods, which align time series and text representations; and prompting-based methods, which directly prompt large language models for forecasting. Our findings reveal that the benefits of multimodality are highly condition-dependent. While we confirm reported gains in some settings, these improvements are not universal across datasets or models. To move beyond empirical observations, we disentangle the effects of model architectural properties and data characteristics, drawing data-agnostic insights that generalize across domains. Our findings highlight that on the modeling side, incorporating text information is most helpful given (1) high-capacity text models, (2) comparatively weaker time series models, and (3) appropriate aligning strategies. On the data side, performance gains are more likely when (4) sufficient training data is available and (5) the text offers complementary predictive signal beyond what is already captured from the time series alone. Our study offers a rigorous, quantitative foundation for understanding when multimodality can be expected to aid forecasting tasks, and reveals that its benefits are neither universal nor always aligned with intuition.