Mitigate the Gap: Investigating Approaches for Improving Cross-Modal Alignment in CLIP

TL;DR

This paper investigates methods to reduce the modality gap in CLIP's embedding space, proposing AlignCLIP, which improves cross-modal alignment and downstream task performance through parameter sharing and intra-modality separation.

Contribution

The paper introduces AlignCLIP, a novel approach that effectively mitigates the modality gap in CLIP, enhancing cross-modal alignment and downstream task performance.

Findings

AlignCLIP reduces the modality gap significantly.

Improved cross-modal alignment leads to better zero-shot and fine-tuning results.

Parameter sharing and intra-modality separation are effective strategies.

Abstract

Contrastive Language--Image Pre-training (CLIP) has manifested remarkable improvements in zero-shot classification and cross-modal vision-language tasks. Yet, from a geometrical point of view, the CLIP embedding space has been found to have a pronounced modality gap. This gap renders the embedding space overly sparse and disconnected, with different modalities being densely distributed in distinct subregions of the hypersphere. In this work, we aim at answering three main questions: 1. Does sharing the parameter space between the multi-modal encoders reduce the modality gap? 2. Can the gap be mitigated by pushing apart the uni-modal embeddings via intra-modality separation? 3. How do these gap reduction approaches affect the downstream performance? We design AlignCLIP, in order to answer these questions and through extensive experiments, we show that AlignCLIP achieves noticeable enhancements in the cross-modal alignment of the embeddings, and thereby, reduces the modality gap, while improving the performance across several zero-shot and fine-tuning downstream evaluations.