ACE Metric: Advection and Convection Evaluation for Accurate Weather Forecasting

TL;DR

The paper introduces the ACE metric, a specialized evaluation tool for weather forecasting models that focuses on advection and convection accuracy, addressing limitations of traditional pixel-wise metrics.

Contribution

It proposes the ACE metric tailored for assessing physical transfer processes in weather models, validated on WeatherBench2 and MovingMNIST datasets.

Findings

ACE effectively evaluates advection and convection accuracy.

ACE correlates better with physical weather phenomena than traditional metrics.

Validated on multiple datasets showing improved assessment of weather models.

Abstract

Recently, data-driven weather forecasting methods have received significant attention for surpassing the RMSE performance of traditional NWP (Numerical Weather Prediction)-based methods. However, data-driven models are tuned to minimize the loss between forecasted data and ground truths, often using pixel-wise loss. This can lead to models that produce blurred outputs, which, despite being significantly different in detail from the actual weather conditions, still demonstrate low RMSE values. Although evaluation metrics from the computer vision field, such as PSNR, SSIM, and FVD, can be used, they are not entirely suitable for weather variables. This is because weather variables exhibit continuous physical changes over time and lack the distinct boundaries of objects typically seen in computer vision images. To resolve these issues, we propose the advection and convection Error (ACE) metric, specifically designed to assess how well models predict advection and convection, which are significant atmospheric transfer methods. We have validated the ACE evaluation metric on the WeatherBench2 and MovingMNIST datasets.