Improving Concept Alignment in Vision-Language Concept Bottleneck Models

TL;DR

This paper improves the alignment of human-defined concepts with visual inputs in vision-language models, enhancing interpretability and trustworthiness through a novel semi-supervised learning approach and class-level interventions.

Contribution

It introduces a Contrastive Semi-Supervised learning method that uses limited labeled data to better align concepts with visual inputs in CLIP models, improving interpretability.

Findings

Significant improvement in concept accuracy (+29.95)

Enhanced classification accuracy (+3.84)

Requires fewer human-annotated labels

Abstract

Concept Bottleneck Models (CBM) map images to human-interpretable concepts before making class predictions. Recent approaches automate CBM construction by prompting Large Language Models (LLMs) to generate text concepts and employing Vision Language Models (VLMs) to score these concepts for CBM training. However, it is desired to build CBMs with concepts defined by human experts rather than LLM-generated ones to make them more trustworthy. In this work, we closely examine the faithfulness of VLM concept scores for such expert-defined concepts in domains like fine-grained bird species and animal classification. Our investigations reveal that VLMs like CLIP often struggle to correctly associate a concept with the corresponding visual input, despite achieving a high classification performance. This misalignment renders the resulting models difficult to interpret and less reliable. To address this issue, we propose a novel Contrastive Semi-Supervised (CSS) learning method that leverages a few labeled concept samples to activate truthful visual concepts and improve concept alignment in the CLIP model. Extensive experiments on three benchmark datasets demonstrate that our method significantly enhances both concept (+29.95) and classification (+3.84) accuracies yet requires only a fraction of human-annotated concept labels. To further improve the classification performance, we introduce a class-level intervention procedure for fine-grained classification problems that identifies the confounding classes and intervenes in their concept space to reduce errors.