Utilizing Strategic Pre-training to Reduce Overfitting: Baguan -- A Pre-trained Weather Forecasting Model

TL;DR

This paper presents Baguan, a pre-trained weather forecasting model that uses a Siamese Autoencoder to mitigate overfitting and improve forecast accuracy with limited data, outperforming traditional methods.

Contribution

Introduction of Baguan, a novel pre-trained weather forecasting model utilizing self-supervised learning to reduce overfitting and enhance performance across various forecasting tasks.

Findings

Baguan outperforms traditional forecasting methods.

Pre-training with locality bias mitigates overfitting.

Robust performance in downstream forecasting tasks.

Abstract

Weather forecasting has long posed a significant challenge for humanity. While recent AI-based models have surpassed traditional numerical weather prediction (NWP) methods in global forecasting tasks, overfitting remains a critical issue due to the limited availability of real-world weather data spanning only a few decades. Unlike fields like computer vision or natural language processing, where data abundance can mitigate overfitting, weather forecasting demands innovative strategies to address this challenge with existing data. In this paper, we explore pre-training methods for weather forecasting, finding that selecting an appropriately challenging pre-training task introduces locality bias, effectively mitigating overfitting and enhancing performance. We introduce Baguan, a novel data-driven model for medium-range weather forecasting, built on a Siamese Autoencoder pre-trained in a self-supervised manner and fine-tuned for different lead times. Experimental results show that Baguan outperforms traditional methods, delivering more accurate forecasts. Additionally, the pre-trained Baguan demonstrates robust overfitting control and excels in downstream tasks, such as subseasonal-to-seasonal (S2S) modeling and regional forecasting, after fine-tuning.