Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification

TL;DR

This paper introduces LaBo, a method that uses GPT-3 to automatically generate and select human-readable concepts for interpretable image classification, achieving high accuracy without manual concept specification.

Contribution

LaBo is the first approach to construct high-performance concept bottleneck models without manual concept definition, leveraging GPT-3 and a novel selection method for diverse, discriminative concepts.

Findings

LaBo outperforms black box linear probes in few-shot classification by 11.7%.

LaBo achieves comparable accuracy to black box models with less data.

LaBo demonstrates broad applicability across 11 diverse datasets.

Abstract

Concept Bottleneck Models (CBM) are inherently interpretable models that factor model decisions into human-readable concepts. They allow people to easily understand why a model is failing, a critical feature for high-stakes applications. CBMs require manually specified concepts and often under-perform their black box counterparts, preventing their broad adoption. We address these shortcomings and are first to show how to construct high-performance CBMs without manual specification of similar accuracy to black box models. Our approach, Language Guided Bottlenecks (LaBo), leverages a language model, GPT-3, to define a large space of possible bottlenecks. Given a problem domain, LaBo uses GPT-3 to produce factual sentences about categories to form candidate concepts. LaBo efficiently searches possible bottlenecks through a novel submodular utility that promotes the selection of discriminative and diverse information. Ultimately, GPT-3's sentential concepts can be aligned to images using CLIP, to form a bottleneck layer. Experiments demonstrate that LaBo is a highly effective prior for concepts important to visual recognition. In the evaluation with 11 diverse datasets, LaBo bottlenecks excel at few-shot classification: they are 11.7% more accurate than black box linear probes at 1 shot and comparable with more data. Overall, LaBo demonstrates that inherently interpretable models can be widely applied at similar, or better, performance than black box approaches.