AIMIP Phase 1: systematic evaluations of AI weather and climate models

TL;DR

AIMIP Phase 1 systematically evaluates AI-based weather and climate models against traditional models, focusing on their ability to simulate historical climate data and responses, with open data for further research.

Contribution

This work introduces a standardized framework and dataset for evaluating AI weather and climate models, highlighting their strengths and limitations compared to conventional models.

Findings

AI models can simulate historical climate and responses similarly to traditional models.

Some AI models underestimate historical warming trends.

AI models diverge in out-of-sample generalization tests.

Abstract

We present the AI weather and climate model intercomparison project (AIMIP), phase 1. Drawing from the rich tradition of intercomparisons in climate model development, we specify a common experiment, output data format, and training constraints (namely, training against historical reanalysis data) for AIMIP Phase 1 models. We aim to identify differences in modeling frameworks and AI architectural choices that influence model behavior, and build trust in AI weather and climate models through open data and evaluation. AIMIP Phase 1 models must simulate the atmosphere given specified historical sea surface temperatures over 1979-2024. We evaluate the models' performance using five major evaluation criteria: biases, trends, response to El Ni\~{n}o-related sea surface temperature anomalies, temporal variability, and out-of-sample generalization tests. We find that the AI models are able to simulate the historical climate and response to forcing as well as a conventional physically-based model, but some AI models underestimate historical warming trends, and their predictions diverge in the out-of-sample generalization tests. We describe the AIMIP Phase 1 dataset that is publicly available for additional evaluations.