Cross-modal Counterfactual Explanations: Uncovering Decision Factors and Dataset Biases in Subjective Classification

TL;DR

This paper introduces DeX, a training-free, cross-modal interpretability framework that generates natural language counterfactual explanations for image privacy decisions, revealing key decision factors and dataset biases.

Contribution

DeX is a novel, flexible approach that leverages image-specific concepts to produce sparse, natural language explanations for subjective image classification decisions.

Findings

DeX outperforms state-of-the-art explanation methods.

It uncovers principal decision factors influencing subjective judgments.

It identifies dataset biases affecting model fairness.

Abstract

Concept-driven counterfactuals explain decisions of classifiers by altering the model predictions through semantic changes. In this paper, we present a novel approach that leverages cross-modal decompositionality and image-specific concepts to create counterfactual scenarios expressed in natural language. We apply the proposed interpretability framework, termed Decompose and Explain (DeX), to the challenging domain of image privacy decisions, which are contextual and subjective. This application enables the quantification of the differential contributions of key scene elements to the model prediction. We identify relevant decision factors via a multi-criterion selection mechanism that considers both image similarity for minimal perturbations and decision confidence to prioritize impactful changes. This approach evaluates and compares diverse explanations, and assesses the interdependency and mutual influence among explanatory properties. By leveraging image-specific concepts, DeX generates image-grounded, sparse explanations, yielding significant improvements over the state of the art. Importantly, DeX operates as a training-free framework, offering high flexibility. Results show that DeX not only uncovers the principal contributing factors influencing subjective decisions, but also identifies underlying dataset biases allowing for targeted mitigation strategies to improve fairness.