Long-Range Distillation: Distilling 10,000 Years of Simulated Climate into Long Timestep AI Weather Models

TL;DR

This paper introduces long-range distillation, a novel training method that uses synthetic climate data generated by a short-timestep model to train long-timestep AI weather models, significantly improving long-range forecast accuracy.

Contribution

The paper presents a new long-range distillation technique that leverages synthetic data from a short-timestep autoregressive model to train effective long-timestep probabilistic weather models.

Findings

Distilled models outperform climatology in perfect-model experiments.

They approach autoregressive teacher skill at long timescales.

Models achieve S2S forecast skill comparable to ECMWF after fine-tuning.

Abstract

Accurate long-range weather forecasting remains a major challenge for AI models, both because errors accumulate over autoregressive rollouts and because reanalysis datasets used for training offer a limited sample of the slow modes of climate variability underpinning predictability. Most AI weather models are autoregressive, producing short lead forecasts that must be repeatedly applied to reach subseasonal-to-seasonal (S2S) or seasonal lead times, often resulting in instability and calibration issues. Long-timestep probabilistic models that generate long-range forecasts in a single step offer an attractive alternative, but training on the 40-year reanalysis record leads to overfitting, suggesting orders of magnitude more training data are required. We introduce long-range distillation, a method that trains a long-timestep probabilistic "student" model to forecast directly at long-range using a huge synthetic training dataset generated by a short-timestep autoregressive "teacher" model. Using the Deep Learning Earth System Model (DLESyM) as the teacher, we generate over 10,000 years of simulated climate to train distilled student models for forecasting across a range of timescales. In perfect-model experiments, the distilled models outperform climatology and approach the skill of their autoregressive teacher while replacing hundreds of autoregressive steps with a single timestep. In the real world, they achieve S2S forecast skill comparable to the ECMWF ensemble forecast after ERA5 fine-tuning. The skill of our distilled models scales with increasing synthetic training data, even when that data is orders of magnitude larger than ERA5. This represents the first demonstration that AI-generated synthetic training data can be used to scale long-range forecast skill.