Discovering Concept Directions from Diffusion-based Counterfactuals via Latent Clustering

TL;DR

This paper introduces CDLC, a scalable and efficient method for extracting concept directions from diffusion-based counterfactuals, enabling interpretable insights in high-stakes AI applications.

Contribution

The paper presents CDLC, a novel clustering-based approach that reduces computational costs and improves the extraction of semantic concepts from diffusion models.

Findings

Reduces storage by ~4.6% and speeds up discovery by ~5.3%.

Aligns with clinically recognized dermoscopic features.

Reveals dataset-specific biases and unknown biomarkers.

Abstract

Concept-based explanations have emerged as an effective approach within Explainable Artificial Intelligence, enabling interpretable insights by aligning model decisions with human-understandable concepts. However, existing methods rely on computationally intensive procedures and struggle to efficiently capture complex, semantic concepts. This work introduces the Concept Directions via Latent Clustering (CDLC), which extracts global, class-specific concept directions by clustering latent difference vectors derived from factual and diffusion-generated counterfactual image pairs. CDLC reduces storage requirements by ~4.6% and accelerates concept discovery by ~5.3% compared to the baseline method, while requiring no GPU for clustering, thereby enabling efficient extraction of multidimensional semantic concepts across latent dimensions. This approach is validated on a real-world skin lesion dataset, demonstrating that the extracted concept directions align with clinically recognized dermoscopic features and, in some cases, reveal dataset-specific biases or unknown biomarkers. These results highlight that CDLC is interpretable, scalable, and applicable across high-stakes domains and diverse data modalities.