ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Scale Multitask Dataset

TL;DR

This paper introduces ITFormer, a novel framework that effectively combines time-series data with natural language for multi-modal question answering, supported by a large-scale dataset and demonstrating improved accuracy with minimal additional training parameters.

Contribution

We propose ITFormer, a new model that bridges time-series encoders with large language models, and release EngineMT-QA, the first large-scale multi-task dataset for temporal-textual QA.

Findings

ITFormer improves QA accuracy over baselines.

The model achieves this with less than 1% additional trainable parameters.

The approach demonstrates effective cross-modal feature fusion.

Abstract

Time-series data are critical in diverse applications, such as industrial monitoring, medical diagnostics, and climate research. However, effectively integrating these high-dimensional temporal signals with natural language for dynamic, interactive tasks remains a significant challenge. To address this, we introduce the Time-Series Question Answering (Time-Series QA) task and release EngineMT-QA, the first large-scale, multi-task, temporal-textual QA dataset designed to capture complex interactions between time-series signals and natural language. Building on this resource, we propose the Instruct Time Transformer (ITFormer), a novel framework that bridges time-series encoders with frozen large language models (LLMs). ITFormer effectively extracts, aligns, and fuses temporal and textual features, achieving a strong improvement in QA accuracy over strong baselines with fewer than 1\% additional trainable parameters. By combining computational efficiency with robust cross-modal modeling, our work establishes a adaptable paradigm for integrating temporal data with natural language, paving the way for new research and applications in multi-modal AI. More details about the project, including datasets and code, are available at: https://pandalin98.github.io/itformer_site/