STEEX: Steering Counterfactual Explanations with Semantics

TL;DR

This paper introduces STEEX, a novel framework for generating plausible, sparse, and region-guided counterfactual explanations for complex, high-quality images using semantic-to-image models, enhancing interpretability in safety-critical applications.

Contribution

It presents a new generative counterfactual explanation method that produces realistic modifications and allows user-guided, region-specific explanations for complex scenes.

Findings

Effective on high-quality portrait datasets

Produces plausible and sparse counterfactuals

Enables user-guided, region-specific explanations

Abstract

As deep learning models are increasingly used in safety-critical applications, explainability and trustworthiness become major concerns. For simple images, such as low-resolution face portraits, synthesizing visual counterfactual explanations has recently been proposed as a way to uncover the decision mechanisms of a trained classification model. In this work, we address the problem of producing counterfactual explanations for high-quality images and complex scenes. Leveraging recent semantic-to-image models, we propose a new generative counterfactual explanation framework that produces plausible and sparse modifications which preserve the overall scene structure. Furthermore, we introduce the concept of "region-targeted counterfactual explanations", and a corresponding framework, where users can guide the generation of counterfactuals by specifying a set of semantic regions of the query image the explanation must be about. Extensive experiments are conducted on challenging datasets including high-quality portraits (CelebAMask-HQ) and driving scenes (BDD100k). Code is available at https://github.com/valeoai/STEEX