ContextCLIP: Contextual Alignment of Image-Text pairs on CLIP visual representations

TL;DR

ContextCLIP introduces a novel framework that enhances image-text alignment in CLIP representations through contextual and contrastive learning, improving zero-shot transfer and classification accuracy across diverse datasets.

Contribution

It presents a new contextual alignment method for CLIP that improves robustness and transfer performance on standard datasets.

Findings

Improved image-text alignment in joint embedding space.

Enhanced zero-shot classification accuracy.

Better qualitative results for text-to-image retrieval.

Abstract

State-of-the-art empirical work has shown that visual representations learned by deep neural networks are robust in nature and capable of performing classification tasks on diverse datasets. For example, CLIP demonstrated zero-shot transfer performance on multiple datasets for classification tasks in a joint embedding space of image and text pairs. However, it showed negative transfer performance on standard datasets, e.g., BirdsNAP, RESISC45, and MNIST. In this paper, we propose ContextCLIP, a contextual and contrastive learning framework for the contextual alignment of image-text pairs by learning robust visual representations on Conceptual Captions dataset. Our framework was observed to improve the image-text alignment by aligning text and image representations contextually in the joint embedding space. ContextCLIP showed good qualitative performance for text-to-image retrieval tasks and enhanced classification accuracy. We evaluated our model quantitatively with zero-shot transfer and fine-tuning experiments on CIFAR-10, CIFAR-100, Birdsnap, RESISC45, and MNIST datasets for classification task.