Cross-Modal Retrieval Meets Inference:Improving Zero-Shot Classification with Cross-Modal Retrieval

TL;DR

This paper introduces X-MoRe, a novel inference method that enhances CLIP's zero-shot classification by leveraging cross-modal retrieval and confidence-based ensemble, improving performance without additional training.

Contribution

Proposes X-MoRe, a new inference technique combining cross-modal retrieval and confidence weighting to boost zero-shot classification performance of CLIP.

Findings

X-MoRe improves zero-shot classification accuracy across multiple tasks.

Utilizing external image-text pairs enhances CLIP's inference without retraining.

The method maintains robustness without additional training data.

Abstract

Contrastive language-image pre-training (CLIP) has demonstrated remarkable zero-shot classification ability, namely image classification using novel text labels. Existing works have attempted to enhance CLIP by fine-tuning on downstream tasks, but these have inadvertently led to performance degradation on unseen classes, thus harming zero-shot generalization. This paper aims to address this challenge by leveraging readily available image-text pairs from an external dataset for cross-modal guidance during inference. To this end, we propose X-MoRe, a novel inference method comprising two key steps: (1) cross-modal retrieval and (2) modal-confidence-based ensemble. Given a query image, we harness the power of CLIP's cross-modal representations to retrieve relevant textual information from an external image-text pair dataset. Then, we assign higher weights to the more reliable modality between the original query image and retrieved text, contributing to the final prediction. X-MoRe demonstrates robust performance across a diverse set of tasks without the need for additional training, showcasing the effectiveness of utilizing cross-modal features to maximize CLIP's zero-shot ability.