Seeing in Words: Learning to Classify through Language Bottlenecks

TL;DR

This paper introduces a vision model that learns to classify images using text-based features, aiming to improve interpretability while maintaining high accuracy on ImageNet.

Contribution

It presents a novel approach where neural networks learn to classify images through language bottlenecks, bridging the gap between interpretability and performance.

Findings

Model effectively classifies ImageNet images using text features

Training such models presents unique challenges

The approach enhances interpretability of neural network predictions

Abstract

Neural networks for computer vision extract uninterpretable features despite achieving high accuracy on benchmarks. In contrast, humans can explain their predictions using succinct and intuitive descriptions. To incorporate explainability into neural networks, we train a vision model whose feature representations are text. We show that such a model can effectively classify ImageNet images, and we discuss the challenges we encountered when training it.