Beyond Average Performance -- exploring regions of deviating performance for black box classification models

TL;DR

This paper introduces two approaches to interpret and describe regions where black box classification models deviate from their average performance, enhancing understanding and accountability in high-stakes applications.

Contribution

It presents novel methods for providing interpretable descriptions of performance deviations in black box classifiers, aiding in model accountability.

Findings

Methods successfully identify regions of deviating performance.

Approaches help warn users about unreliable model predictions.

Enhances interpretability of complex models in critical applications.

Abstract

Machine learning models are becoming increasingly popular in different types of settings. This is mainly caused by their ability to achieve a level of predictive performance that is hard to match by human experts in this new era of big data. With this usage growth comes an increase of the requirements for accountability and understanding of the models' predictions. However, the degree of sophistication of the most successful models (e.g. ensembles, deep learning) is becoming a large obstacle to this endeavour as these models are essentially black boxes. In this paper we describe two general approaches that can be used to provide interpretable descriptions of the expected performance of any black box classification model. These approaches are of high practical relevance as they provide means to uncover and describe in an interpretable way situations where the models are expected to have a performance that deviates significantly from their average behaviour. This may be of critical relevance for applications where costly decisions are driven by the predictions of the models, as it can be used to warn end users against the usage of the models in some specific cases.