Opening the Black Box of the Radiation Belt Machine Learning Model

TL;DR

This paper demonstrates that a high-accuracy machine learning model for Earth's radiation belt electron flux can be interpreted using DeepSHAP, revealing physically meaningful feature importance during specific geomagnetic events.

Contribution

The study introduces a method to interpret a neural network model of radiation belt electrons using DeepSHAP, bridging the gap between model accuracy and physical understanding.

Findings

DeepSHAP successfully explains the model's predictions.

Feature importances align with physical phenomena.

Model captures key dynamics during geomagnetic events.

Abstract

Many Machine Learning (ML) systems, especially neural networks, are fundamentally regarded as black boxes since it is difficult to grasp how they function once they have been trained. Here, we tackle the issue of the interpretability of a high-accuracy ML model created to model the flux of Earth's radiation belt electrons. The Outer RadIation belt Electron Neural net model (ORIENT) uses only solar wind conditions and geomagnetic indices as input. Using the Deep SHAPley additive explanations (DeepSHAP) method, we show that the `black box' ORIENT model can be successfully explained. Two significant electron flux enhancement events observed by Van Allen Probes during the storm interval of 17 to 18 March 2013 and non storm interval of 19 to 20 September 2013 are investigated using the DeepSHAP method. The results show that the feature importances calculated from the purely data driven ORIENT model identify physically meaningful behavior consistent with current physical understanding.