Towards Long-Context Time Series Foundation Models

TL;DR

This paper enhances time series foundation models to effectively handle long, multivariate data by comparing context expansion techniques and introducing a novel compressive memory mechanism, improving their practical applicability in domains like healthcare.

Contribution

It systematically compares context expansion methods and introduces a new compressive memory mechanism for encoder-only time series models, enabling better modeling of intra-variate dependencies.

Findings

Improved modeling of long, multivariate time series data.

Enhanced performance of MOMENT models with multivariate context.

Demonstrated effectiveness in healthcare and other domains.

Abstract

Time series foundation models have shown impressive performance on a variety of tasks, across a wide range of domains, even in zero-shot settings. However, most of these models are designed to handle short univariate time series as an input. This limits their practical use, especially in domains such as healthcare with copious amounts of long and multivariate data with strong temporal and intra-variate dependencies. Our study bridges this gap by cataloging and systematically comparing various context expansion techniques from both language and time series domains, and introducing a novel compressive memory mechanism to allow encoder-only TSFMs to effectively model intra-variate dependencies. We demonstrate the benefits of our approach by imbuing MOMENT, a recent family of multi-task time series foundation models, with the multivariate context.