Towards Faithful Multimodal Concept Bottleneck Models

TL;DR

This paper introduces f-CBM, a multimodal concept bottleneck model that jointly improves concept detection and reduces leakage, achieving a better balance between accuracy and interpretability across image and text datasets.

Contribution

f-CBM is a novel framework that combines a leakage loss and an expressive prediction head to enhance faithfulness in multimodal concept bottleneck models.

Findings

f-CBM achieves superior trade-offs between accuracy, concept detection, and leakage reduction.

The model is versatile, working effectively on both image and text datasets.

It outperforms existing methods by jointly addressing detection and leakage issues.

Abstract

Concept Bottleneck Models (CBMs) are interpretable models that route predictions through a layer of human-interpretable concepts. While widely studied in vision and, more recently, in NLP, CBMs remain largely unexplored in multimodal settings. For their explanations to be faithful, CBMs must satisfy two conditions: concepts must be properly detected, and concept representations must encode only their intended semantics, without smuggling extraneous task-relevant or inter-concept information into final predictions, a phenomenon known as leakage. Existing approaches treat concept detection and leakage mitigation as separate problems, and typically improve one at the expense of predictive accuracy. In this work, we introduce f-CBM, a faithful multimodal CBM framework built on a vision-language backbone that jointly targets both aspects through two complementary strategies: a differentiable leakage loss to mitigate leakage, and a Kolmogorov-Arnold Network prediction head that provides sufficient expressiveness to improve concept detection. Experiments demonstrate that f-CBM achieves the best trade-off between task accuracy, concept detection, and leakage reduction, while applying seamlessly to both image and text or text-only datasets, making it versatile across modalities.