Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations

TL;DR

This paper introduces VH-CBM, a hybrid concept bottleneck model that combines vision-language models and minimal human annotations to improve concept quality and interpretability.

Contribution

VH-CBM employs a Gaussian Process to effectively propagate limited annotations, enhancing concept accuracy and interpretability over existing VLM-guided CBMs.

Findings

VH-CBM outperforms VLM-guided CBMs with as little as 1% annotations.

VH-CBM achieves better concept calibration.

Supports active learning for efficient annotation.

Abstract

Concept-bottleneck models (CBMs) are neural classifiers that compute predictions from high-level concepts extracted from the input. CBMs ensure stakeholders can understand the concepts -- and the predictions they entail -- by learning these from concept-level annotations, which are however seldom available. Recent CBM architectures work around this issue by obtaining annotations from Vision-Language Models (VLMs). While greatly broadening applicability, doing so can yield lower quality concepts and therefore less interpretable models. We strike for a middle ground by introducing Vision-plus-Human-guided CBM (VH-CBM), a hybrid approach that exploits both VLMs and a small amount of dense annotations. VH-CBM employs a Gaussian Process in the VLM's embedding space, which captures useful global information about the target domain, to propagate the expert's supervision to any target data point. Our empirical evaluation shows how VH-CBM predicts more accurate concepts than VLM-guided CBMs even when annotating as little as 1% of the data, while sporting better concept calibration and supporting active learning.