Thoth: Mid-Training Bridges LLMs to Time Series Understanding

TL;DR

Thoth introduces a novel mid-training approach for large language models to understand and reason about time series data, significantly improving their performance on time series tasks and benchmarks.

Contribution

This work presents Thoth, the first family of mid-trained LLMs with general-purpose time series understanding, using a new corpus and benchmark for evaluation.

Findings

Thoth outperforms base models and advanced LLMs on time series question answering.

Mid-training with Book-of-Thoth enhances temporal reasoning capabilities.

Thoth performs well under data scarcity when fine-tuned.

Abstract

Large Language Models (LLMs) have demonstrated remarkable success in general-purpose reasoning. However, they still struggle to understand and reason about time series data, which limits their effectiveness in decision-making scenarios that depend on temporal dynamics. In this paper, we propose Thoth, the first family of mid-trained LLMs with general-purpose time series understanding capabilities. As a pivotal intermediate stage, mid-training achieves task- and domain-agnostic alignment between time series and natural language, for which we construct Book-of-Thoth, a high-quality, time-series-centric mid-training corpus. Book-of-Thoth enables both time-series-to-text and text-to-time-series generation, equipping LLMs with a foundational grasp of temporal patterns. To better evaluate advanced reasoning capabilities, we further present KnoTS, a novel benchmark of knowledge-intensive time series understanding, designed for joint reasoning over temporal patterns and domain knowledge. Extensive experiments demonstrate that mid-training with Book-of-Thoth enables Thoth to significantly outperform its base model and advanced LLMs across a range of time series question answering benchmarks. Moreover, Thoth exhibits superior capabilities when fine-tuned under data scarcity, underscoring the effectiveness of mid-training for time series understanding. Code is available at: https://github.com/thuml/Thoth.