Improving Zero-shot Generalization and Robustness of Multi-modal Models

TL;DR

This paper identifies the causes of low top-1 accuracy in multi-modal models like CLIP, proposes a post-hoc uncertainty detection method, and enhances accuracy by leveraging WordNet hierarchy, significantly improving performance without extra training.

Contribution

It introduces a simple, scalable method to identify uncertain predictions and improves accuracy by incorporating semantic hierarchy into prompts, addressing the top-1 gap in zero-shot models.

Findings

Improved top-1 accuracy by 17.13% on uncertain images

Enhanced overall accuracy by 3.6% on ImageNet validation set

Method outperforms max logit baseline in selective prediction

Abstract

Multi-modal image-text models such as CLIP and LiT have demonstrated impressive performance on image classification benchmarks and their zero-shot generalization ability is particularly exciting. While the top-5 zero-shot accuracies of these models are very high, the top-1 accuracies are much lower (over 25% gap in some cases). We investigate the reasons for this performance gap and find that many of the failure cases are caused by ambiguity in the text prompts. First, we develop a simple and efficient zero-shot post-hoc method to identify images whose top-1 prediction is likely to be incorrect, by measuring consistency of the predictions w.r.t. multiple prompts and image transformations. We show that our procedure better predicts mistakes, outperforming the popular max logit baseline on selective prediction tasks. Next, we propose a simple and efficient way to improve accuracy on such uncertain images by making use of the WordNet hierarchy; specifically we augment the original class by incorporating its parent and children from the semantic label hierarchy, and plug the augmentation into text prompts. We conduct experiments on both CLIP and LiT models with five different ImageNet-based datasets. For CLIP, our method improves the top-1 accuracy by 17.13% on the uncertain subset and 3.6% on the entire ImageNet validation set. We also show that our method improves across ImageNet shifted datasets, four other datasets, and other model architectures such as LiT. The proposed method is hyperparameter-free, requires no additional model training and can be easily scaled to other large multi-modal architectures. Code is available at https://github.com/gyhandy/Hierarchy-CLIP.