Trustworthy predictive distributions for rare events via diagnostic transport maps

TL;DR

This paper introduces diagnostic transport maps to recalibrate and improve the reliability of predictive distributions for rare events, providing real-time diagnostics and enhanced accuracy in complex forecasting scenarios.

Contribution

It proposes a novel method using diagnostic transport maps to adjust and validate predictive distributions, especially in low-frequency and out-of-distribution regimes.

Findings

Improved calibration of predictive distributions for rare events.

Enhanced detection of model failures and biases.

Better predictive performance in tropical cyclone intensity forecasting.

Abstract

Forecast systems in science and technology are increasingly moving beyond point prediction toward methods that produce full predictive distributions of future outcomes y, conditional on high-dimensional and complex sequences of inputs x. However, even when forecast systems provide a full predictive distribution, the result is rarely calibrated with respect to all x and y. The estimated density can be especially unreliable in low-frequency or out-of-distribution regimes, where accurate uncertainty quantification and a means for human experts to verify results are most needed to establish trust in models. In this paper, we take an initial predictive distribution as given and treat it as a useful but potentially misspecified base model. WE then introduce diagnostic transport maps, covariate-dependent probability-to-probability maps that quantify how the base model's probabilities should be adjusted to better match the true conditional distribution of calibration data. At deployment, these maps provide the user with real-time local diagnostics that reveal where the model fails and how it fails (including bias, dispersion, skewness, and tail errors), while also producing a recalibrated predictive distribution through a simple composition with the base model. We apply diagnostic transport maps to short-term tropical cyclone intensity forecasting and show that an easy-to-fit parametric version identifies evolutionary modes associated with local miscalibration and improves the predictive performance for rare events, including 24-hour rapid intensity change, as compared to the operational forecasts of the National Hurricane Center.