Text-to-Image Models for Counterfactual Explanations: a Black-Box Approach

TL;DR

This paper introduces TIME, a black-box method using text-to-image models to generate counterfactual explanations for classifiers, requiring only the image and prediction, without access to model internals.

Contribution

TIME is a novel black-box approach that leverages bias injection into Stable Diffusion to produce counterfactual explanations without needing classifier details.

Findings

TIME generates effective counterfactuals comparable to existing methods.

The approach requires only image and prediction, not model parameters.

Empirical results validate the method's robustness and effectiveness.

Abstract

This paper addresses the challenge of generating Counterfactual Explanations (CEs), involving the identification and modification of the fewest necessary features to alter a classifier's prediction for a given image. Our proposed method, Text-to-Image Models for Counterfactual Explanations (TIME), is a black-box counterfactual technique based on distillation. Unlike previous methods, this approach requires solely the image and its prediction, omitting the need for the classifier's structure, parameters, or gradients. Before generating the counterfactuals, TIME introduces two distinct biases into Stable Diffusion in the form of textual embeddings: the context bias, associated with the image's structure, and the class bias, linked to class-specific features learned by the target classifier. After learning these biases, we find the optimal latent code applying the classifier's predicted class token and regenerate the image using the target embedding as conditioning, producing the counterfactual explanation. Extensive empirical studies validate that TIME can generate explanations of comparable effectiveness even when operating within a black-box setting.