Less is More: Unlocking Specialization of Time Series Foundation Models via Structured Pruning

TL;DR

This paper introduces a structured pruning approach for Time Series Foundation Models that enhances their adaptation to specific forecasting tasks, leading to improved performance over larger, unpruned models and surpassing specialized baselines.

Contribution

The paper proposes a novel structured pruning method to regularize fine-tuning of TSFMs, enabling more effective task-specific adaptation and improved forecasting accuracy.

Findings

Pruned TSFMs outperform original models in fine-tuning tasks.

Pruning enables TSFMs to surpass specialized models and achieve state-of-the-art results.

Empirical evidence across multiple models and benchmarks supports the effectiveness of the approach.

Abstract

Scaling laws motivate the development of Time Series Foundation Models (TSFMs) that pre-train vast parameters and achieve remarkable zero-shot forecasting performance. Surprisingly, even after fine-tuning, TSFMs cannot consistently outperform smaller, specialized models trained on full-shot downstream data. A key question is how to realize effective adaptation of TSFMs for a target forecasting task. Through empirical studies on various TSFMs, the pre-trained models often exhibit inherent sparsity and redundancy in computation, suggesting that TSFMs have learned to activate task-relevant network substructures to accommodate diverse forecasting tasks. To preserve this valuable prior knowledge, we propose a structured pruning method to regularize the subsequent fine-tuning process by focusing it on a more relevant and compact parameter space. Extensive experiments on seven TSFMs and six benchmarks demonstrate that fine-tuning a smaller, pruned TSFM significantly improves forecasting performance compared to fine-tuning original models. This prune-then-finetune paradigm often enables TSFMs to achieve state-of-the-art performance and surpass strong specialized baselines. Source code is made publicly available at https://github.com/SJTU-DMTai/Prune-then-Finetune.