Explaining in Diffusion: Explaining a Classifier Through Hierarchical Semantics with Text-to-Image Diffusion Models

TL;DR

DiffEx uses text-to-image diffusion models to provide hierarchical, detailed explanations of classifier decisions across simple and complex concepts, surpassing traditional GAN-based methods in semantic coverage.

Contribution

The paper introduces DiffEx, a novel diffusion-based explainability method that offers hierarchical, fine-grained insights into classifier decisions, unlike traditional GAN-based approaches.

Findings

DiffEx covers a broader spectrum of semantics.

It provides hierarchical, detailed explanations.

It outperforms GAN-based methods in semantic coverage.

Abstract

Classifiers are important components in many computer vision tasks, serving as the foundational backbone of a wide variety of models employed across diverse applications. However, understanding the decision-making process of classifiers remains a significant challenge. We propose DiffEx, a novel method that leverages the capabilities of text-to-image diffusion models to explain classifier decisions. Unlike traditional GAN-based explainability models, which are limited to simple, single-concept analyses and typically require training a new model for each classifier, our approach can explain classifiers that focus on single concepts (such as faces or animals) as well as those that handle complex scenes involving multiple concepts. DiffEx employs vision-language models to create a hierarchical list of semantics, allowing users to identify not only the overarching semantic influences on classifiers (e.g., the 'beard' semantic in a facial classifier) but also their sub-types, such as 'goatee' or 'Balbo' beard. Our experiments demonstrate that DiffEx is able to cover a significantly broader spectrum of semantics compared to its GAN counterparts, providing a hierarchical tool that delivers a more detailed and fine-grained understanding of classifier decisions.