Validity in machine learning for extreme event attribution

TL;DR

This paper evaluates the validity of machine learning methods in extreme event attribution for climate-related disasters, highlighting key challenges and proposing more robust analytical approaches.

Contribution

It identifies major threats to validity in ML-based EEA, including sensitivity to design choices and distribution shift, and proposes improved evaluation metrics and diagnostics.

Findings

Event attribution estimates are highly sensitive to algorithmic choices.

Standard performance metrics do not reliably indicate attribution accuracy.

Distribution shift significantly reduces predictive performance.

Abstract

Extreme event attribution (EEA), an approach for assessing the extent to which disasters are caused by climate change, is crucial for informing climate policy and legal proceedings. Machine learning is increasingly used for EEA by modeling rare weather events otherwise too complex or computationally intensive to model using traditional simulation methods. However, the validity of using machine learning in this context remains unclear, particularly as high-stakes machine learning applications in general are criticized for inherent bias and lack of robustness. Here we use machine learning and simulation analyses to evaluate EEA in the context of California wildfire data from 2003-2020. We identify three major threats to validity: (1) individual event attribution estimates are highly sensitive to algorithmic design choices; (2) common performance metrics like area under the ROC curve or Brier score are not strongly correlated with attribution error, facilitating suboptimal model selection; and (3) distribution shift -- changes in temperature across climate scenarios -- substantially degrades predictive performance. To address these challenges, we propose a more valid and robust attribution analysis based on aggregate machine learning estimates, using an additional metric -- mean calibration error -- to assess model performance, and using subgroup and propensity diagnostics to assess distribution shift.