Explaining and visualizing black-box models through counterfactual paths

TL;DR

This paper introduces a new XAI method using counterfactual paths generated by feature permutations to explain and visualize black-box models, especially in knowledge graphs, with demonstrated effectiveness on synthetic and medical data.

Contribution

It presents a novel counterfactual path-based approach for explaining black-box models, integrating domain knowledge and enhancing interpretability and visualization capabilities.

Findings

Effective in identifying influential feature permutations

Applicable to knowledge graphs with domain knowledge

Demonstrated on synthetic and medical datasets

Abstract

Explainable AI (XAI) is an increasingly important area of machine learning research, which aims to make black-box models transparent and interpretable. In this paper, we propose a novel approach to XAI that uses the so-called counterfactual paths generated by conditional permutations of features. The algorithm measures feature importance by identifying sequential permutations of features that most influence changes in model predictions. It is particularly suitable for generating explanations based on counterfactual paths in knowledge graphs incorporating domain knowledge. Counterfactual paths introduce an additional graph dimension to current XAI methods in both explaining and visualizing black-box models. Experiments with synthetic and medical data demonstrate the practical applicability of our approach.