AdaCBM: An Adaptive Concept Bottleneck Model for Explainable and Accurate Diagnosis

TL;DR

This paper introduces AdaCBM, an adaptive module that improves medical diagnosis accuracy and explainability by bridging CLIP and CBM models, addressing transfer learning limitations in medical imaging.

Contribution

The paper proposes an innovative adaptive module between CLIP and CBM, enhancing classification performance while maintaining explainability in medical diagnosis tasks.

Findings

Enhanced classification accuracy in medical imaging.

Preserved explainability through concept bottleneck models.

Effective bridging of source and target domains.

Abstract

The integration of vision-language models such as CLIP and Concept Bottleneck Models (CBMs) offers a promising approach to explaining deep neural network (DNN) decisions using concepts understandable by humans, addressing the black-box concern of DNNs. While CLIP provides both explainability and zero-shot classification capability, its pre-training on generic image and text data may limit its classification accuracy and applicability to medical image diagnostic tasks, creating a transfer learning problem. To maintain explainability and address transfer learning needs, CBM methods commonly design post-processing modules after the bottleneck module. However, this way has been ineffective. This paper takes an unconventional approach by re-examining the CBM framework through the lens of its geometrical representation as a simple linear classification system. The analysis uncovers that post-CBM fine-tuning modules merely rescale and shift the classification outcome of the system, failing to fully leverage the system's learning potential. We introduce an adaptive module strategically positioned between CLIP and CBM to bridge the gap between source and downstream domains. This simple yet effective approach enhances classification performance while preserving the explainability afforded by the framework. Our work offers a comprehensive solution that encompasses the entire process, from concept discovery to model training, providing a holistic recipe for leveraging the strengths of GPT, CLIP, and CBM.