S^2-KD: Semantic-Spectral Knowledge Distillation Spatiotemporal Forecasting

TL;DR

S^2-KD introduces a novel semantic-spectral knowledge distillation framework that leverages multimodal teacher models to improve spatiotemporal forecasting accuracy, especially in complex, long-term scenarios.

Contribution

It unifies semantic priors with spectral representations in KD, enabling lightweight models to learn richer, causally-informed features without textual input at inference.

Findings

Outperforms state-of-the-art methods on WeatherBench and TaxiBJ+ benchmarks.

Enhances long-horizon and non-stationary forecasting accuracy.

Enables simple models to learn semantically coherent and spectrally accurate predictions.

Abstract

Spatiotemporal forecasting often relies on computationally intensive models to capture complex dynamics. Knowledge distillation (KD) has emerged as a key technique for creating lightweight student models, with recent advances like frequency-aware KD successfully preserving spectral properties (i.e., high-frequency details and low-frequency trends). However, these methods are fundamentally constrained by operating on pixel-level signals, leaving them blind to the rich semantic and causal context behind the visual patterns. To overcome this limitation, we introduce S^2-KD, a novel framework that unifies Semantic priors with Spectral representations for distillation. Our approach begins by training a privileged, multimodal teacher model. This teacher leverages textual narratives from a Large Multimodal Model (LMM) to reason about the underlying causes of events, while its architecture simultaneously decouples spectral components in its latent space. The core of our framework is a new distillation objective that transfers this unified semantic-spectral knowledge into a lightweight, vision-only student. Consequently, the student learns to make predictions that are not only spectrally accurate but also semantically coherent, without requiring any textual input or architectural overhead at inference. Extensive experiments on benchmarks like WeatherBench and TaxiBJ+ show that S^2-KD significantly boosts the performance of simple student models, enabling them to outperform state-of-the-art methods, particularly in long-horizon and complex non-stationary scenarios.