Deducing neighborhoods of classes from a fitted model

TL;DR

This paper introduces a new interpretability method called Quantile Shift Method (QSM) that helps understand class neighborhoods in complex classification models by analyzing prediction changes after feature manipulations, demonstrated on medical and real data.

Contribution

The paper presents QSM, a novel interpretability technique that reveals class neighborhoods through quantile shifts, enhancing understanding of complex models' decision boundaries.

Findings

QSM effectively identifies class neighborhoods in models.

Application to medical data demonstrates practical usefulness.

Visualization with chordgraphs aids interpretation.

Abstract

In todays world the request for very complex models for huge data sets is rising steadily. The problem with these models is that by raising the complexity of the models, it gets much harder to interpret them. The growing field of \emph{interpretable machine learning} tries to make up for the lack of interpretability in these complex (or even blackbox-)models by using specific techniques that can help to understand those models better. In this article a new kind of interpretable machine learning method is presented, which can help to understand the partitioning of the feature space into predicted classes in a classification model using quantile shifts. To illustrate in which situations this quantile shift method (QSM) could become beneficial, it is applied to a theoretical medical example and a real data example. Basically, real data points (or specific points of interest) are used and the changes of the prediction after slightly raising or decreasing specific features are observed. By comparing the predictions before and after the manipulations, under certain conditions the observed changes in the predictions can be interpreted as neighborhoods of the classes with regard to the manipulated features. Chordgraphs are used to visualize the observed changes.