On the Feasibility of Vision-Language Models for Time-Series Classification

TL;DR

This paper explores using Vision-Language Models for time-series classification by combining graphical and numerical data, demonstrating competitive results with minimal fine-tuning and proposing a scalable training pipeline.

Contribution

It introduces a novel method integrating graphical data representations with numerical data for time-series classification using VLMs, and develops a scalable training pipeline.

Findings

VLMs achieve competitive results after two epochs of fine-tuning.

Graphical representations enhance contextual understanding of time-series data.

The approach is effective for both univariate and multivariate data.

Abstract

We build upon time-series classification by leveraging the capabilities of Vision Language Models (VLMs). We find that VLMs produce competitive results after two or less epochs of fine-tuning. We develop a novel approach that incorporates graphical data representations as images in conjunction with numerical data. This approach is rooted in the hypothesis that graphical representations can provide additional contextual information that numerical data alone may not capture. Additionally, providing a graphical representation can circumvent issues such as limited context length faced by LLMs. To further advance this work, we implemented a scalable end-to-end pipeline for training on different scenarios, allowing us to isolate the most effective strategies for transferring learning capabilities from LLMs to Time Series Classification (TSC) tasks. Our approach works with univariate and multivariate time-series data. In addition, we conduct extensive and practical experiments to show how this approach works for time-series classification and generative labels.