COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training

TL;DR

COSMOS introduces a novel self-distillation framework with cross-attention and multi-modal augmentations, improving vision-language pre-training by capturing comprehensive image and text information for better downstream task performance.

Contribution

The paper proposes COSMOS, a self-distillation method with cross-attention and multi-modal augmentations, enhancing vision-language models beyond contrastive loss limitations.

Findings

Outperforms previous baselines on zero-shot retrieval, classification, and segmentation tasks.

Surpasses larger dataset CLIP models in visual perception and understanding.

Effective integration of cross-attention and multi-modal views improves cross-modal representations.

Abstract

Vision-Language Models (VLMs) trained with contrastive loss have achieved significant advancements in various vision and language tasks. However, the global nature of the contrastive loss makes VLMs focus predominantly on foreground objects, neglecting other crucial information in the image, which limits their effectiveness in downstream tasks. To address these challenges, we propose COSMOS: CrOSs-MOdality Self-distillation for vision-language pre-training that integrates a novel text-cropping strategy and cross-attention module into a self-supervised learning framework. We create global and local views of images and texts (i.e., multi-modal augmentations), which are essential for self-distillation in VLMs. We further introduce a cross-attention module, enabling COSMOS to learn comprehensive cross-modal representations optimized via a cross-modality self-distillation loss. COSMOS consistently outperforms previous strong baselines on various zero-shot downstream tasks, including retrieval, classification, and semantic segmentation. Additionally, it surpasses CLIP-based models trained on larger datasets in visual perception and contextual understanding tasks. Code is available at https://github.com/ExplainableML/cosmos.