When are Post-hoc Conceptual Explanations Identifiable?

TL;DR

This paper establishes conditions under which post-hoc concept discovery in embedding spaces can be provably reliable, connecting classical methods and proposing new approaches for dependent concepts, with significant empirical improvements.

Contribution

It introduces the concept of identifiability in concept discovery, linking it to classical methods and proposing novel approaches for dependent concepts, ensuring provable recovery of known concepts.

Findings

Proposed methods outperform competitors by up to 29% in alignment with ground truth.

Established formal conditions for reliable, label-free concept discovery.

Connected classical PCA/ICA methods to concept recovery under specific distributions.

Abstract

Interest in understanding and factorizing learned embedding spaces through conceptual explanations is steadily growing. When no human concept labels are available, concept discovery methods search trained embedding spaces for interpretable concepts like object shape or color that can provide post-hoc explanations for decisions. Unlike previous work, we argue that concept discovery should be identifiable, meaning that a number of known concepts can be provably recovered to guarantee reliability of the explanations. As a starting point, we explicitly make the connection between concept discovery and classical methods like Principal Component Analysis and Independent Component Analysis by showing that they can recover independent concepts under non-Gaussian distributions. For dependent concepts, we propose two novel approaches that exploit functional compositionality properties of image-generating processes. Our provably identifiable concept discovery methods substantially outperform competitors on a battery of experiments including hundreds of trained models and dependent concepts, where they exhibit up to 29 % better alignment with the ground truth. Our results highlight the strict conditions under which reliable concept discovery without human labels can be guaranteed and provide a formal foundation for the domain. Our code is available online.