LG-CAV: Train Any Concept Activation Vector with Language Guidance

TL;DR

LG-CAV leverages pre-trained vision-language models to train concept activation vectors without labeled data, enabling explainability and model correction across diverse datasets and architectures.

Contribution

Proposes a novel language-guided approach to train CAVs without labeled data using vision-language models and introduces a model correction technique called activation sample reweighting.

Findings

LG-CAV outperforms previous CAV methods in quality across datasets.

The model correction method achieves state-of-the-art results.

Applicable to various architectures and datasets.

Abstract

Concept activation vector (CAV) has attracted broad research interest in explainable AI, by elegantly attributing model predictions to specific concepts. However, the training of CAV often necessitates a large number of high-quality images, which are expensive to curate and thus limited to a predefined set of concepts. To address this issue, we propose Language-Guided CAV (LG-CAV) to harness the abundant concept knowledge within the certain pre-trained vision-language models (e.g., CLIP). This method allows training any CAV without labeled data, by utilizing the corresponding concept descriptions as guidance. To bridge the gap between vision-language model and the target model, we calculate the activation values of concept descriptions on a common pool of images (probe images) with vision-language model and utilize them as language guidance to train the LG-CAV. Furthermore, after training high-quality LG-CAVs related to all the predicted classes in the target model, we propose the activation sample reweighting (ASR), serving as a model correction technique, to improve the performance of the target model in return. Experiments on four datasets across nine architectures demonstrate that LG-CAV achieves significantly superior quality to previous CAV methods given any concept, and our model correction method achieves state-of-the-art performance compared to existing concept-based methods. Our code is available at https://github.com/hqhQAQ/LG-CAV.