Post-hoc Self-explanation of CNNs

TL;DR

This paper proposes a post-hoc explanation method for CNNs using k-means classifiers and concept-based maps, balancing interpretability and performance.

Contribution

It introduces a formal framework for k-means-based explanations and leverages spatial consistency for concept maps, improving interpretability of CNNs.

Findings

Using less compressed features improves semantic fidelity.

Replacing the final layer with k-means maintains performance.

Concept maps provide meaningful explanations.

Abstract

Although standard Convolutional Neural Networks (CNNs) can be mathematically reinterpreted as Self-Explainable Models (SEMs), their built-in prototypes do not on their own accurately represent the data. Replacing the final linear layer with a $k$-means-based classifier addresses this limitation without compromising performance. This work introduces a common formalization of $k$-means-based post-hoc explanations for the classifier, the encoder's final output (B4), and combinations of intermediate feature activations. The latter approach leverages the spatial consistency of convolutional receptive fields to generate concept-based explanation maps, which are supported by gradient-free feature attribution maps. Empirical evaluation with a ResNet34 shows that using shallower, less compressed feature activations, such as those from the last three blocks (B234), results in a trade-off between semantic fidelity and a slight reduction in predictive performance.