Frequency-Aligned Knowledge Distillation for Lightweight Spatiotemporal Forecasting

TL;DR

This paper introduces SDKD, a spectral frequency-aligned knowledge distillation method that enhances lightweight spatiotemporal forecasting models by transferring multi-scale spectral features from complex teachers, improving accuracy and efficiency.

Contribution

The paper proposes a novel frequency-aligned knowledge distillation strategy that effectively transfers multi-scale spectral features from teacher to student models for spatiotemporal forecasting.

Findings

Achieves up to 81.3% reduction in MSE on Navier-Stokes dataset.

Significantly improves forecasting accuracy by capturing both high-frequency and low-frequency components.

Reduces training time and memory consumption compared to traditional methods.

Abstract

Spatiotemporal forecasting tasks, such as traffic flow, combustion dynamics, and weather forecasting, often require complex models that suffer from low training efficiency and high memory consumption. This paper proposes a lightweight framework, Spectral Decoupled Knowledge Distillation (termed SDKD), which transfers the multi-scale spatiotemporal representations from a complex teacher model to a more efficient lightweight student network. The teacher model follows an encoder-latent evolution-decoder architecture, where its latent evolution module decouples high-frequency details and low-frequency trends using convolution and Transformer (global low-frequency modeler). However, the multi-layer convolution and deconvolution structures result in slow training and high memory usage. To address these issues, we propose a frequency-aligned knowledge distillation strategy, which extracts multi-scale spectral features from the teacher's latent space, including both high and low frequency components, to guide the lightweight student model in capturing both local fine-grained variations and global evolution patterns. Experimental results show that SDKD significantly improves performance, achieving reductions of up to 81.3% in MSE and in MAE 52.3% on the Navier-Stokes equation dataset. The framework effectively captures both high-frequency variations and long-term trends while reducing computational complexity. Our codes are available at https://github.com/itsnotacie/SDKD