Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data

TL;DR

This paper introduces a theoretical framework and a novel three-step method, C^3, to improve cross-modal learning from uni-modal data by addressing the modality gap in contrastive representation spaces, leading to state-of-the-art results.

Contribution

It provides a theoretical analysis of the multi-modal contrastive space and proposes C^3, a method to bridge the modality gap, enhancing cross-modal task performance from uni-modal data.

Findings

Achieves state-of-the-art zero-shot captioning results

Improves cross-modal learning effectiveness

Validates the theoretical analysis with empirical results

Abstract

Building cross-modal applications is challenging due to limited paired multi-modal data. Recent works have shown that leveraging a pre-trained multi-modal contrastive representation space enables cross-modal tasks to be learned from uni-modal data. This is based on the assumption that contrastive optimization makes embeddings from different modalities interchangeable. However, this assumption is under-explored due to the poorly understood geometry of the multi-modal contrastive space, where a modality gap exists. In our study, we provide a theoretical explanation of this space's geometry and introduce a three-step method, $C^3$ (Connect, Collapse, Corrupt), to bridge the modality gap, enhancing the interchangeability of embeddings. Our $C^3$ method significantly improves cross-modal learning from uni-modal data, achieving state-of-the-art results on zero-shot image / audio / video captioning and text-to-image generation.