WeatherFormer: A Pretrained Encoder Model for Learning Robust Weather Representations from Small Datasets

TL;DR

WeatherFormer is a transformer-based model pretrained on extensive satellite data that learns robust weather representations from small datasets, significantly improving predictions in agriculture and epidemiology.

Contribution

It introduces WeatherFormer, a novel pretrained transformer encoder that effectively models weather dynamics from limited data, with innovative spatiotemporal encoding and pretraining strategies.

Findings

Achieves state-of-the-art soybean yield prediction

Improves influenza forecasting accuracy

Demonstrates effectiveness across multiple weather-dependent domains

Abstract

This paper introduces WeatherFormer, a transformer encoder-based model designed to learn robust weather features from minimal observations. It addresses the challenge of modeling complex weather dynamics from small datasets, a bottleneck for many prediction tasks in agriculture, epidemiology, and climate science. WeatherFormer was pretrained on a large pretraining dataset comprised of 39 years of satellite measurements across the Americas. With a novel pretraining task and fine-tuning, WeatherFormer achieves state-of-the-art performance in county-level soybean yield prediction and influenza forecasting. Technical innovations include a unique spatiotemporal encoding that captures geographical, annual, and seasonal variations, adapting the transformer architecture to continuous weather data, and a pretraining strategy to learn representations that are robust to missing weather features. This paper for the first time demonstrates the effectiveness of pretraining large transformer encoder models for weather-dependent applications across multiple domains.