Probabilistically Plausible Counterfactual Explanations with Normalizing Flows

TL;DR

PPCEF is a novel method that generates high-quality, probabilistically plausible counterfactual explanations using normalizing flows, offering faster computation and better alignment with data distribution for interpretability.

Contribution

The paper introduces PPCEF, a unified probabilistic framework leveraging normalizing flows for generating plausible counterfactuals efficiently without assuming specific distribution families.

Findings

Outperforms existing methods in plausibility and speed

Effectively captures complex data distributions in high-dimensional settings

Produces counterfactuals that improve model interpretability and trust

Abstract

We present PPCEF, a novel method for generating probabilistically plausible counterfactual explanations (CFs). PPCEF advances beyond existing methods by combining a probabilistic formulation that leverages the data distribution with the optimization of plausibility within a unified framework. Compared to reference approaches, our method enforces plausibility by directly optimizing the explicit density function without assuming a particular family of parametrized distributions. This ensures CFs are not only valid (i.e., achieve class change) but also align with the underlying data's probability density. For that purpose, our approach leverages normalizing flows as powerful density estimators to capture the complex high-dimensional data distribution. Furthermore, we introduce a novel loss that balances the trade-off between achieving class change and maintaining closeness to the original instance while also incorporating a probabilistic plausibility term. PPCEF's unconstrained formulation allows for efficient gradient-based optimization with batch processing, leading to orders of magnitude faster computation compared to prior methods. Moreover, the unconstrained formulation of PPCEF allows for the seamless integration of future constraints tailored to specific counterfactual properties. Finally, extensive evaluations demonstrate PPCEF's superiority in generating high-quality, probabilistically plausible counterfactual explanations in high-dimensional tabular settings. This makes PPCEF a powerful tool for not only interpreting complex machine learning models but also for improving fairness, accountability, and trust in AI systems.