SAFE: A Novel Approach to AI Weather Evaluation through Stratified Assessments of Forecasts over Earth

TL;DR

SAFE introduces a stratified evaluation framework for weather forecasts, revealing disparities in model performance across geospatial and socio-economic attributes, thus enabling more equitable and location-specific model assessments.

Contribution

It presents a novel stratified assessment package that evaluates weather forecast accuracy across diverse Earth attributes, highlighting performance disparities and promoting fairness in model evaluation.

Findings

Models show significant performance disparities across regions and attributes.

SAFE uncovers biases in existing weather prediction models.

Benchmarking reveals which models are most fair across different strata.

Abstract

The dominant paradigm in machine learning is to assess model performance based on average loss across all samples in some test set. This amounts to averaging performance geospatially across the Earth in weather and climate settings, failing to account for the non-uniform distribution of human development and geography. We introduce Stratified Assessments of Forecasts over Earth (SAFE), a package for elucidating the stratified performance of a set of predictions made over Earth. SAFE integrates various data domains to stratify by different attributes associated with geospatial gridpoints: territory (usually country), global subregion, income, and landcover (land or water). This allows us to examine the performance of models for each individual stratum of the different attributes (e.g., the accuracy in every individual country). To demonstrate its importance, we utilize SAFE to benchmark a zoo of state-of-the-art AI-based weather prediction models, finding that they all exhibit disparities in forecasting skill across every attribute. We use this to seed a benchmark of model forecast fairness through stratification at different lead times for various climatic variables. By moving beyond globally-averaged metrics, we for the first time ask: where do models perform best or worst, and which models are most fair? To support further work in this direction, the SAFE package is open source and available at https://github.com/N-Masi/safe