Olivia: Harmonizing Time Series Foundation Models with Power Spectral Density

TL;DR

Olivia introduces a novel time series foundation model that uses spectral domain harmonization via PSDs to improve transferability and performance across diverse datasets, achieving state-of-the-art results.

Contribution

The paper proposes Olivia, a new foundation model with a spectral harmonization mechanism called Harmonizer, enhancing transferability and performance in time series forecasting.

Findings

Olivia outperforms existing models on large-scale benchmarks.

Harmonizer effectively reduces dataset mismatches in spectral domain.

Olivia demonstrates strong zero-shot, few-shot, and full-shot forecasting capabilities.

Abstract

Time series foundation models rely on large-scale pretraining over diverse datasets across domains, yet their heterogeneity in temporal patterns could hinder the effectiveness of training and learning transferable time series representations. Inspired a fundamental concept, normalized power spectral density (PSD) in signal processing, we assume harmonizing datasets via PSDs in the spectral domain could reduce mismatches and enhance pretraining. We then go beyond the direct intractable minimization optimization and innovatively reformulate it as a principled harmonization approach. Specifically, we propose Harmonizer, a module that reshapes spectral structures and implicitly harmonizing PSDs across datasets, which theoretically corresponds to a shared reparameterization of second-order temporal correlations. Our theoretical analysis further reveals token interactions with Harmonizer can be efficiently mediated by a compact set of resonators, motivating a HarmonicAttention design that performs self-attention in a low-dimensional interaction space. Then, we propose Olivia, a novel time series foundation model built upon these harmonization mechanisms. Extensive experiments on two large-scale benchmarks (TSLib and GIFT-Eval) and extra 6 datasets from GluonTS, demonstrate Olivia consistently achieves state-of-the-art performance under zero-shot, few-shot, and full-shot forecasting scenarios. Our code is available at https://github.com/TSTS13/Olivia.