Probabilistic Forecasting using Deep Generative Models

TL;DR

This paper explores using deep generative models to improve probabilistic weather forecasting by reducing memory and computation costs compared to traditional analog ensemble methods.

Contribution

It introduces a novel approach that replaces large historical datasets with generative models, enabling faster and more memory-efficient probabilistic forecasts.

Findings

Generative models significantly reduce memory requirements.

Forecast generation time is decreased to constant time.

The method maintains forecast accuracy while improving efficiency.

Abstract

The Analog Ensemble (AnEn) method tries to estimate the probability distribution of the future state of the atmosphere with a set of past observations that correspond to the best analogs of a deterministic Numerical Weather Prediction (NWP). This model post-processing method has been successfully used to improve the forecast accuracy for several weather-related applications including air quality, and short-term wind and solar power forecasting, to name a few. In order to provide a meaningful probabilistic forecast, the AnEn method requires storing a historical set of past predictions and observations in memory for a period of at least several months and spanning the seasons relevant for the prediction of interest. Although the memory and computing costs of the AnEn method are less expensive than using a brute-force dynamical ensemble approach, for a large number of stations and large datasets, the amount of memory required for AnEn can easily become prohibitive. Furthermore, in order to find the best analogs associated with a certain prediction produced by a NWP model, the current approach requires searching over the entire dataset by applying a certain metric. This approach requires applying the metric over the entire historical dataset, which may take a substantial amount of time. In this work, we investigate an alternative way to implement the AnEn method using deep generative models. By doing so, a generative model can entirely or partially replace the dataset of pairs of predictions and observations, reducing the amount of memory required to produce the probabilistic forecast by several orders of magnitude. Furthermore, the generative model can generate a meaningful set of analogs associated with a certain forecast in constant time without performing any search, saving a considerable amount of time even in the presence of huge historical datasets.