Assessing reliability of explanations in unbalanced datasets: a use-case on the occurrence of frost events

TL;DR

This paper explores the robustness of XAI explanations in unbalanced datasets, specifically for frost event prediction, proposing a simple evaluation method for explanation reliability in high-risk, imbalanced scenarios.

Contribution

It introduces a novel evaluation approach for XAI explanation reliability in unbalanced datasets, focusing on the minority class and on-manifold neighbor generation.

Findings

The proposed method assesses explanation consistency effectively.

Application to frost event data demonstrates its practical utility.

Highlights challenges of explanation robustness in imbalanced high-risk datasets.

Abstract

The usage of eXplainable Artificial Intelligence (XAI) methods has become essential in practical applications, given the increasing deployment of Artificial Intelligence (AI) models and the legislative requirements put forward in the latest years. A fundamental but often underestimated aspect of the explanations is their robustness, a key property that should be satisfied in order to trust the explanations. In this study, we provide some preliminary insights on evaluating the reliability of explanations in the specific case of unbalanced datasets, which are very frequent in high-risk use-cases, but at the same time considerably challenging for both AI models and XAI methods. We propose a simple evaluation focused on the minority class (i.e. the less frequent one) that leverages on-manifold generation of neighbours, explanation aggregation and a metric to test explanation consistency. We present a use-case based on a tabular dataset with numerical features focusing on the occurrence of frost events.