CLIP-QDA: An Explainable Concept Bottleneck Model

TL;DR

CLIP-QDA introduces an explainable, fast image classification method based on a concept bottleneck approach using a Mixture of Gaussians to enhance interpretability, achieving competitive accuracy and providing both local and global explanations.

Contribution

This work presents CLIP-QDA, a novel concept bottleneck model that leverages a Mixture of Gaussians for interpretability and combines performance with explainability in image classification.

Findings

Achieves similar accuracy to state-of-the-art CBMs when MoG assumption holds.

Provides faster explanations compared to existing XAI methods.

Offers both local and global interpretability through its architecture.

Abstract

In this paper, we introduce an explainable algorithm designed from a multi-modal foundation model, that performs fast and explainable image classification. Drawing inspiration from CLIP-based Concept Bottleneck Models (CBMs), our method creates a latent space where each neuron is linked to a specific word. Observing that this latent space can be modeled with simple distributions, we use a Mixture of Gaussians (MoG) formalism to enhance the interpretability of this latent space. Then, we introduce CLIP-QDA, a classifier that only uses statistical values to infer labels from the concepts. In addition, this formalism allows for both local and global explanations. These explanations come from the inner design of our architecture, our work is part of a new family of greybox models, combining performances of opaque foundation models and the interpretability of transparent models. Our empirical findings show that in instances where the MoG assumption holds, CLIP-QDA achieves similar accuracy with state-of-the-art methods CBMs. Our explanations compete with existing XAI methods while being faster to compute.