Climatological benchmarking of AI-generated tropical cyclones

TL;DR

This paper benchmarks AI-generated tropical cyclones against observed data, evaluating their ability to replicate key climatological features and physical consistency, revealing strengths and biases of current AI models for storm simulation.

Contribution

It provides a comprehensive comparison of AI weather models Pangu-Weather and Aurora in simulating tropical cyclone climatology and identifies systematic biases for future improvements.

Findings

AI models reproduce storm track density and speed well

Aurora outperforms Pangu in simulating storm intensity

Both models overestimate storm size, especially for extreme events

Abstract

This study presents a comprehensive climatological benchmarking of tropical cyclones (TCs) generated by AI-based global weather prediction models. Using all TC events from the North Atlantic and Western Pacific basins between 2020 and 2025, we assess the ability of two AI models (Pangu-Weather and Aurora) to reproduce observed TC track density, climatology of storm characteristics, and physical consistency with TC theory. By comparing AI-simulated TCs with ERA5 reanalysis, we benchmark the distributions of intensity, size, forward speed, and evaluate the model's ability to credibly simulate extratropical transition. Results show that both Pangu and Aurora perform well in reproducing storm track density, forward speed distribution, and outer size distribution. Aurora shows an improved performance in simulating storm intensity compared to Pangu, with less bias in the distribution of minimum central pressure and maximum wind speed. However, both models overestimate the distribution of storm inner size (radius of maximum winds), especially for extreme events. AI models capture the relative frequency and temporal evolution of extratropical transition patterns with reasonable accuracy. The AI-simulated TCs are also less likely to conform to gradient wind balance compared to ERA5, indicating that the AI TCs may not be physically realistic in many cases. This benchmarking identifies systematic biases that can guide future corrections and support extended applications of AI models for TC hazard and risk assessment. Our work establishes a foundation for future studies using AI weather models in the context of TC climatological and hazard research.