Investigating the fidelity of explainable artificial intelligence methods for applications of convolutional neural networks in geoscience

TL;DR

This study evaluates the accuracy and limitations of various explainable AI methods in interpreting CNN decisions in geoscience, highlighting issues that could distort understanding and guiding better practices.

Contribution

It provides an intercomparison of popular XAI methods applied to CNNs in geoscience, revealing their strengths and weaknesses through benchmark and real-world climate prediction applications.

Findings

XAI methods face issues like gradient shattering and sign attribution errors.

Some XAI methods cannot distinguish zero input contributions.

The analysis highlights the need for cautious application of XAI in geoscience.

Abstract

Convolutional neural networks (CNNs) have recently attracted great attention in geoscience due to their ability to capture non-linear system behavior and extract predictive spatiotemporal patterns. Given their black-box nature however, and the importance of prediction explainability, methods of explainable artificial intelligence (XAI) are gaining popularity as a means to explain the CNN decision-making strategy. Here, we establish an intercomparison of some of the most popular XAI methods and investigate their fidelity in explaining CNN decisions for geoscientific applications. Our goal is to raise awareness of the theoretical limitations of these methods and gain insight into the relative strengths and weaknesses to help guide best practices. The considered XAI methods are first applied to an idealized attribution benchmark, where the ground truth of explanation of the network is known a priori, to help objectively assess their performance. Secondly, we apply XAI to a climate-related prediction setting, namely to explain a CNN that is trained to predict the number of atmospheric rivers in daily snapshots of climate simulations. Our results highlight several important issues of XAI methods (e.g., gradient shattering, inability to distinguish the sign of attribution, ignorance to zero input) that have previously been overlooked in our field and, if not considered cautiously, may lead to a distorted picture of the CNN decision-making strategy. We envision that our analysis will motivate further investigation into XAI fidelity and will help towards a cautious implementation of XAI in geoscience, which can lead to further exploitation of CNNs and deep learning for prediction problems.