CWCL: Cross-Modal Transfer with Continuously Weighted Contrastive Loss

TL;DR

This paper introduces CWCL, a novel continuous similarity-based contrastive loss for cross-modal zero-shot transfer, improving alignment and performance across image-text and speech-text tasks over existing methods.

Contribution

The paper proposes the Continuously Weighted Contrastive Loss (CWCL), a new loss function that models similarity as a continuous measure, enhancing cross-modal representation alignment.

Findings

Achieves 5-8% improvement in zero-shot image classification.

Achieves 20-30% improvement in zero-shot speech-to-intent classification.

Outperforms existing methods across multiple models, datasets, and modalities.

Abstract

This paper considers contrastive training for cross-modal 0-shot transfer wherein a pre-trained model in one modality is used for representation learning in another domain using pairwise data. The learnt models in the latter domain can then be used for a diverse set of tasks in a zero-shot way, similar to ``Contrastive Language-Image Pre-training (CLIP)'' and ``Locked-image Tuning (LiT)'' that have recently gained considerable attention. Most existing works for cross-modal representation alignment (including CLIP and LiT) use the standard contrastive training objective, which employs sets of positive and negative examples to align similar and repel dissimilar training data samples. However, similarity amongst training examples has a more continuous nature, thus calling for a more `non-binary' treatment. To address this, we propose a novel loss function called Continuously Weighted Contrastive Loss (CWCL) that employs a continuous measure of similarity. With CWCL, we seek to align the embedding space of one modality with another. Owing to the continuous nature of similarity in the proposed loss function, these models outperform existing methods for 0-shot transfer across multiple models, datasets and modalities. Particularly, we consider the modality pairs of image-text and speech-text and our models achieve 5-8% (absolute) improvement over previous state-of-the-art methods in 0-shot image classification and 20-30% (absolute) improvement in 0-shot speech-to-intent classification and keyword classification.