VITRO: Vocabulary Inversion for Time-series Representation Optimization

TL;DR

VITRO introduces a novel vocabulary inversion method that learns dataset-specific time series embeddings, significantly improving long-term forecasting performance by bridging the gap between language model vocabularies and continuous time series data.

Contribution

The paper proposes VITRO, a new approach adapting textual inversion to learn time series-specific vocabularies, enhancing representation and forecasting accuracy.

Findings

VITRO achieves state-of-the-art long-term forecasting results.

Learnable pseudo-word embeddings outperform existing vocabularies.

VITRO effectively bridges language and time series data representations.

Abstract

Although LLMs have demonstrated remarkable capabilities in processing and generating textual data, their pre-trained vocabularies are ill-suited for capturing the nuanced temporal dynamics and patterns inherent in time series. The discrete, symbolic nature of natural language tokens, which these vocabularies are designed to represent, does not align well with the continuous, numerical nature of time series data. To address this fundamental limitation, we propose VITRO. Our method adapts textual inversion optimization from the vision-language domain in order to learn a new time series per-dataset vocabulary that bridges the gap between the discrete, semantic nature of natural language and the continuous, numerical nature of time series data. We show that learnable time series-specific pseudo-word embeddings represent time series data better than existing general language model vocabularies, with VITRO-enhanced methods achieving state-of-the-art performance in long-term forecasting across most datasets.