OccamVTS: Distilling Vision Models to 1% Parameters for Time Series Forecasting

TL;DR

OccamVTS introduces a method to distill large vision models into ultra-lightweight networks with only 1% of parameters, significantly improving time series forecasting accuracy by removing unnecessary semantic features.

Contribution

The paper presents OccamVTS, a novel knowledge distillation framework that effectively reduces vision model parameters for time series tasks, enhancing performance and efficiency.

Findings

Achieves state-of-the-art results with 1% parameters

Improves accuracy by removing semantic noise

Excels in few-shot and zero-shot scenarios

Abstract

Time series forecasting is fundamental to diverse applications, with recent approaches leverage large vision models (LVMs) to capture temporal patterns through visual representations. We reveal that while vision models enhance forecasting performance, 99% of their parameters are unnecessary for time series tasks. Through cross-modal analysis, we find that time series align with low-level textural features but not high-level semantics, which can impair forecasting accuracy. We propose OccamVTS, a knowledge distillation framework that extracts only the essential 1% of predictive information from LVMs into lightweight networks. Using pre-trained LVMs as privileged teachers, OccamVTS employs pyramid-style feature alignment combined with correlation and feature distillation to transfer beneficial patterns while filtering out semantic noise. Counterintuitively, this aggressive parameter reduction improves accuracy by eliminating overfitting to irrelevant visual features while preserving essential temporal patterns. Extensive experiments across multiple benchmark datasets demonstrate that OccamVTS consistently achieves state-of-the-art performance with only 1% of the original parameters, particularly excelling in few-shot and zero-shot scenarios.