Multi-Modal Forecaster: Jointly Predicting Time Series and Textual Data

TL;DR

This paper introduces a new multimodal dataset combining time series and textual data from climate and healthcare, and proposes a joint forecasting model that surprisingly underperforms existing baselines, highlighting challenges in multimodal forecasting.

Contribution

The paper provides the first curated multimodal dataset for time series and text, and develops a joint forecasting model, revealing difficulties in multimodal prediction tasks.

Findings

The Hybrid-MMF model did not outperform baselines.

The dataset enables multimodal forecasting research.

Challenges in joint time series and text prediction are significant.

Abstract

Current forecasting approaches are largely unimodal and ignore the rich textual data that often accompany the time series due to lack of well-curated multimodal benchmark dataset. In this work, we develop TimeText Corpus (TTC), a carefully curated, time-aligned text and time dataset for multimodal forecasting. Our dataset is composed of sequences of numbers and text aligned to timestamps, and includes data from two different domains: climate science and healthcare. Our data is a significant contribution to the rare selection of available multimodal datasets. We also propose the Hybrid Multi-Modal Forecaster (Hybrid-MMF), a multimodal LLM that jointly forecasts both text and time series data using shared embeddings. However, contrary to our expectations, our Hybrid-MMF model does not outperform existing baselines in our experiments. This negative result highlights the challenges inherent in multimodal forecasting. Our code and data are available at https://github.com/Rose-STL-Lab/Multimodal_ Forecasting.