Embracing Diversity: Interpretable Zero-shot classification beyond one vector per class

TL;DR

This paper introduces a novel zero-shot classification method that models intra-class diversity using inferred attributes, improving accuracy and interpretability without retraining, especially for diverse and atypical object instances.

Contribution

It proposes a new approach to encode class diversity with inferred attributes, moving beyond single-vector representations in zero-shot classification, enhancing performance and transparency.

Findings

Outperforms standard zero-shot classifiers across multiple datasets.

Scales efficiently to many attributes, improving accuracy for atypical instances.

Provides interpretable explanations for each inference.

Abstract

Vision-language models enable open-world classification of objects without the need for any retraining. While this zero-shot paradigm marks a significant advance, even today's best models exhibit skewed performance when objects are dissimilar from their typical depiction. Real world objects such as pears appear in a variety of forms -- from diced to whole, on a table or in a bowl -- yet standard VLM classifiers map all instances of a class to a \it{single vector based on the class label}. We argue that to represent this rich diversity within a class, zero-shot classification should move beyond a single vector. We propose a method to encode and account for diversity within a class using inferred attributes, still in the zero-shot setting without retraining. We find our method consistently outperforms standard zero-shot classification over a large suite of datasets encompassing hierarchies, diverse object states, and real-world geographic diversity, as well finer-grained datasets where intra-class diversity may be less prevalent. Importantly, our method is inherently interpretable, offering faithful explanations for each inference to facilitate model debugging and enhance transparency. We also find our method scales efficiently to a large number of attributes to account for diversity -- leading to more accurate predictions for atypical instances. Finally, we characterize a principled trade-off between overall and worst class accuracy, which can be tuned via a hyperparameter of our method. We hope this work spurs further research into the promise of zero-shot classification beyond a single class vector for capturing diversity in the world, and building transparent AI systems without compromising performance.