Concept-wise Attention for Fine-grained Concept Bottleneck Models

TL;DR

The paper introduces CoAt-CBM, a novel framework that enhances fine-grained concept alignment and interpretability in concept bottleneck models using learnable visual queries and contrastive optimization.

Contribution

It proposes a new method employing concept-wise visual queries and contrastive loss to improve concept alignment and interpretability in CBMs.

Findings

CoAt-CBM outperforms state-of-the-art methods in experiments.

The approach achieves adaptive fine-grained image-concept alignment.

It enhances interpretability of concept predictions.

Abstract

Recently impressive performance has been achieved in Concept Bottleneck Models (CBM) by utilizing the image-text alignment learned by a large pre-trained vision-language model (i.e. CLIP). However, there exist two key limitations in concept modeling. Existing methods often suffer from pre-training biases, manifested as granularity misalignment or reliance on structural priors. Moreover, fine-tuning with Binary Cross-Entropy (BCE) loss treats each concept independently, which ignores mutual exclusivity among concepts, leading to suboptimal alignment. To address these limitations, we propose Concept-wise Attention for Fine-grained Concept Bottleneck Models (CoAt-CBM), a novel framework that achieves adaptive fine-grained image-concept alignment and high interpretability. Specifically, CoAt-CBM employs learnable concept-wise visual queries to adaptively obtain fine-grained concept-wise visual embeddings, which are then used to produce a concept score vector. Then, a novel concept contrastive optimization guides the model to handle the relative importance of the concept scores, enabling concept predictions to faithfully reflect the image content and improved alignment. Extensive experiments demonstrate that CoAt-CBM consistently outperforms state-of-the-art methods. The codes will be available upon acceptance.