Using proper divergence functions to evaluate climate models

TL;DR

This paper emphasizes the importance of using proper divergence functions, like the integrated quadratic distance and Kullback-Leibler divergence, for evaluating climate models to ensure optimal and truthful assessments.

Contribution

It advocates for the use of proper divergence functions in climate model evaluation and demonstrates their application with a comparison of fifteen climate models.

Findings

Proper divergence functions lead to more reliable model evaluations.

Integrated quadratic distance and Kullback-Leibler divergence are particularly effective.

Some commonly used divergences are not proper and may mislead evaluations.

Abstract

It has been argued persuasively that, in order to evaluate climate models, the probability distributions of model output need to be compared to the corresponding empirical distributions of observed data. Distance measures between probability distributions, also called divergence functions, can be used for this purpose. We contend that divergence functions ought to be proper, in the sense that acting on modelers' true beliefs is an optimal strategy. Score divergences that derive from proper scoring rules are proper, with the integrated quadratic distance and the Kullback-Leibler divergence being particularly attractive choices. Other commonly used divergences fail to be proper. In an illustration, we evaluate and rank simulations from fifteen climate models for temperature extremes in a comparison to re-analysis data.