Explainability in CNN Models By Means of Z-Scores

TL;DR

This paper introduces a simple Z-score based method to interpret CNN output layers, revealing input importance and feature scale relevance, demonstrated on Arctic sea ice prediction with SAR and MWR data.

Contribution

It presents a novel, straightforward approach to explain CNN outputs by comparing them to logistic regression, aiding interpretability and reducing analysis complexity.

Findings

MWR components are more influential than SAR in predictions

The method identifies feature importance at different scales

Simplifies analysis by reducing components for visualization

Abstract

This paper explores the similarities of output layers in Neural Networks (NNs) with logistic regression to explain importance of inputs by Z-scores. The network analyzed, a network for fusion of Synthetic Aperture Radar (SAR) and Microwave Radiometry (MWR) data, is applied to prediction of arctic sea ice. With the analysis the importance of MWR relative to SAR is found to favor MWR components. Further, as the model represents image features at different scales, the relative importance of these are as well analyzed. The suggested methodology offers a simple and easy framework for analyzing output layer components and can reduce the number of components for further analysis with e.g. common NN visualization methods.