Towards a Deeper Understanding of Concept Bottleneck Models Through End-to-End Explanation

TL;DR

This paper investigates how Concept Bottleneck Models (CBMs) relate input features to concepts and final predictions, evaluating explanation methods and proposing relevance proportion as a measure of concept importance.

Contribution

It provides a detailed analysis of relevance distribution in CBMs, compares explanation techniques, and introduces relevance proportion as a new interpretability metric.

Findings

LRP shows lower average distance to ground truth than IG.

Relevance is mainly distributed among input features for concept prediction.

Proposed relevance proportion as a measure of concept importance.

Abstract

Concept Bottleneck Models (CBMs) first map raw input(s) to a vector of human-defined concepts, before using this vector to predict a final classification. We might therefore expect CBMs capable of predicting concepts based on distinct regions of an input. In doing so, this would support human interpretation when generating explanations of the model's outputs to visualise input features corresponding to concepts. The contribution of this paper is threefold: Firstly, we expand on existing literature by looking at relevance both from the input to the concept vector, confirming that relevance is distributed among the input features, and from the concept vector to the final classification where, for the most part, the final classification is made using concepts predicted as present. Secondly, we report a quantitative evaluation to measure the distance between the maximum input feature relevance and the ground truth location; we perform this with the techniques, Layer-wise Relevance Propagation (LRP), Integrated Gradients (IG) and a baseline gradient approach, finding LRP has a lower average distance than IG. Thirdly, we propose using the proportion of relevance as a measurement for explaining concept importance.