Flexible Concept Bottleneck Model

TL;DR

The paper introduces FCBM, a flexible concept bottleneck model that enables dynamic concept adaptation and seamless integration of new concepts without retraining, enhancing interpretability and adaptability in vision-language tasks.

Contribution

We propose a hypernetwork-based approach with a learnable sparsemax module for dynamic concept replacement and selection, improving flexibility over existing VLM-based CBMs.

Findings

Achieves accuracy comparable to state-of-the-art models with similar concepts.

Generalizes well to unseen concepts with minimal fine-tuning.

Supports complete replacement of concept sets without retraining.

Abstract

Concept bottleneck models (CBMs) improve neural network interpretability by introducing an intermediate layer that maps human-understandable concepts to predictions. Recent work has explored the use of vision-language models (VLMs) to automate concept selection and annotation. However, existing VLM-based CBMs typically require full model retraining when new concepts are involved, which limits their adaptability and flexibility in real-world scenarios, especially considering the rapid evolution of vision-language foundation models. To address these issues, we propose Flexible Concept Bottleneck Model (FCBM), which supports dynamic concept adaptation, including complete replacement of the original concept set. Specifically, we design a hypernetwork that generates prediction weights based on concept embeddings, allowing seamless integration of new concepts without retraining the entire model. In addition, we introduce a modified sparsemax module with a learnable temperature parameter that dynamically selects the most relevant concepts, enabling the model to focus on the most informative features. Extensive experiments on five public benchmarks demonstrate that our method achieves accuracy comparable to state-of-the-art baselines with a similar number of effective concepts. Moreover, the model generalizes well to unseen concepts with just a single epoch of fine-tuning, demonstrating its strong adaptability and flexibility.