Unleashing The Power of Pre-Trained Language Models for Irregularly Sampled Time Series

TL;DR

This paper introduces ISTS-PLM, a novel framework leveraging pre-trained language models to effectively analyze irregularly sampled time series, addressing a significant gap in existing methods focused mainly on regular sampling.

Contribution

It proposes a unified PLM-based framework with time-aware and variable-aware models specifically designed for irregularly sampled time series analysis.

Findings

Achieves state-of-the-art results across multiple tasks

Effective in healthcare, biomechanics, and climate science domains

Outperforms existing methods on benchmark datasets

Abstract

Pre-trained Language Models (PLMs), such as ChatGPT, have significantly advanced the field of natural language processing. This progress has inspired a series of innovative studies that explore the adaptation of PLMs to time series analysis, intending to create a unified foundation model that addresses various time series analytical tasks. However, these efforts predominantly focus on Regularly Sampled Time Series (RSTS), neglecting the unique challenges posed by Irregularly Sampled Time Series (ISTS), which are characterized by uneven sampling intervals and prevalent missing data. To bridge this gap, this work takes the first step in exploring the potential of PLMs for ISTS analysis. We begin by investigating the effect of various methods for representing ISTS, aiming to maximize the efficacy of PLMs in the analysis. Furthermore, we propose a unified PLM-based framework, named ISTS-PLM, to address diverse ISTS analytical tasks. It integrates novel time-aware and variable-aware PLMs tailored to tackle the intractable intra- and inter-time series modeling in ISTS. Finally, extensive experiments on a comprehensive benchmark demonstrate that the ISTS-PLM, utilizing a structured and effective series-based representation for ISTS, consistently achieves state-of-the-art performance across various analytical tasks, such as classification, interpolation, extrapolation, few-shot and zero-shot learning scenarios, spanning scientific domains like healthcare, biomechanics, and climate science.