On the explainability of max-plus neural networks

TL;DR

This paper explores the interpretability of max-plus neural networks, demonstrating their decision process can be traced and proposing a pixel fragility measure that outperforms some existing explanation methods.

Contribution

It introduces a method to interpret max-plus neural networks and a pixel fragility measure that effectively explains model decisions.

Findings

The model's decision process can be traced to a single neuron.

The pixel fragility measure correlates well with classification changes.

The explanation method outperforms SHAP and Integrated Gradient on PneumoniaMnist.

Abstract

We investigate the explanability properties of the recently proposed linear-min-max neural networks. At initialization, they can be interpreted as k-medoids with the infinity norm as a distance. Then, they are trained using subgradient descent to better fit the data. The model has been shown to be a universal approximator. Yet, we can trace the decision process because a single most activated neuron is responsible for the value of the output. Using this property, we designed a pixel fragility measure that determines whether changes to a single pixel may be responsible to a change in the classification output. Experiments on the PneumoniaMnist dataset show that this explanation for the output of the neural network compares favorably to SHAP and Integrated Gradient.