V2C-CBM: Building Concept Bottlenecks with Vision-to-Concept Tokenizer

TL;DR

This paper introduces V2C-CBM, a novel concept bottleneck model that constructs visual concepts directly from multimodal models using a vision-to-concept tokenizer, enhancing interpretability and accuracy without extensive annotations.

Contribution

The paper proposes a new method to build concept bottlenecks directly from multimodal models using a vision-to-concept tokenizer, reducing annotation needs and improving performance.

Findings

V2C-CBM matches or outperforms LLM-supervised CBMs on benchmarks.

The approach is training efficient and highly interpretable.

Constructs explicit visual concepts from unlabeled images.

Abstract

Concept Bottleneck Models (CBMs) offer inherent interpretability by initially translating images into human-comprehensible concepts, followed by a linear combination of these concepts for classification. However, the annotation of concepts for visual recognition tasks requires extensive expert knowledge and labor, constraining the broad adoption of CBMs. Recent approaches have leveraged the knowledge of large language models to construct concept bottlenecks, with multimodal models like CLIP subsequently mapping image features into the concept feature space for classification. Despite this, the concepts produced by language models can be verbose and may introduce non-visual attributes, which hurts accuracy and interpretability. In this study, we investigate to avoid these issues by constructing CBMs directly from multimodal models. To this end, we adopt common words as base concept vocabulary and leverage auxiliary unlabeled images to construct a Vision-to-Concept (V2C) tokenizer that can explicitly quantize images into their most relevant visual concepts, thus creating a vision-oriented concept bottleneck tightly coupled with the multimodal model. This leads to our V2C-CBM which is training efficient and interpretable with high accuracy. Our V2C-CBM has matched or outperformed LLM-supervised CBMs on various visual classification benchmarks, validating the efficacy of our approach.