Empowering Time Series Analysis with Large-Scale Multimodal Pretraining

TL;DR

This paper introduces a novel multimodal pretraining paradigm and a large-scale dataset for time series analysis, leading to a foundation model that significantly improves zero-shot forecasting and anomaly detection.

Contribution

It pioneers a comprehensive multimodal pretraining approach and develops MM-TS, the first large-scale multimodal time series dataset, enhancing model generalization across domains.

Findings

HORAI achieves state-of-the-art zero-shot performance in forecasting.

The multimodal dataset MM-TS spans six domains with up to one billion points.

The proposed model effectively fuses heterogeneous modalities for improved analysis.

Abstract

While existing time series foundation models primarily rely on large-scale unimodal pretraining, they lack complementary modalities to enhance time series understanding. Building multimodal foundation models is a natural next step, but it faces key challenges: 1) lack of a unified multimodal pretraining paradigm and large-scale multimodal corpora for time series analysis; 2) how to effectively integrate heterogeneous modalities and enhance model generalization. To address these challenges, we take an early step toward multimodal foundation models for time series analysis. We first propose a multimodal pretraining paradigm that leverages time series with endogenous modalities (derived images and text) and exogenous knowledge (real-world news), providing a comprehensive multi-view perspective for time series analysis. To support this, we develop an automated data construction pipeline to curate MM-TS, the first large-scale multimodal time series dataset spanning six domains, with up to one billion points. Then we propose HORAI, a frequency-enhanced multimodal foundation model. It integrates two core components: the Frequency-enhanced Cross-Modality Encoder and the Time-Frequency Decoder, designed to effectively fuse multimodal features and enhance model generalization across modalities and domains. After pretraining on MM-TS, HORAI achieves state-of-the-art zero-shot performance on time series forecasting and anomaly detection tasks, demonstrating strong generalization.