Evaluating Large Language Models on Time Series Feature Understanding: A Comprehensive Taxonomy and Benchmark

TL;DR

This paper introduces a comprehensive framework and dataset for evaluating large language models' ability to understand and interpret various features of time series data across multiple domains.

Contribution

It presents a detailed taxonomy of time series features and a new benchmark dataset to systematically assess LLMs' time series understanding capabilities.

Findings

LLMs perform well on certain time series features

Model understanding varies with data formatting and series length

Sensitivity to data presentation affects model performance

Abstract

Large Language Models (LLMs) offer the potential for automatic time series analysis and reporting, which is a critical task across many domains, spanning healthcare, finance, climate, energy, and many more. In this paper, we propose a framework for rigorously evaluating the capabilities of LLMs on time series understanding, encompassing both univariate and multivariate forms. We introduce a comprehensive taxonomy of time series features, a critical framework that delineates various characteristics inherent in time series data. Leveraging this taxonomy, we have systematically designed and synthesized a diverse dataset of time series, embodying the different outlined features, each accompanied by textual descriptions. This dataset acts as a solid foundation for assessing the proficiency of LLMs in comprehending time series. Our experiments shed light on the strengths and limitations of state-of-the-art LLMs in time series understanding, revealing which features these models readily comprehend effectively and where they falter. In addition, we uncover the sensitivity of LLMs to factors including the formatting of the data, the position of points queried within a series and the overall time series length.