How do Cross-View and Cross-Modal Alignment Affect Representations in Contrastive Learning?

TL;DR

This paper investigates how cross-view and cross-modal alignment in contrastive learning influence visual representations, revealing that cross-modal alignment emphasizes depth cues over color and texture, affecting downstream tasks.

Contribution

It provides a comprehensive analysis of the effects of cross-view and cross-modal alignment on learned representations across multiple datasets and tasks.

Findings

Cross-modal alignment discards color and texture information.

Depth cues from pretraining improve depth prediction.

Cross-modal alignment yields more robust encoders for certain tasks.

Abstract

Various state-of-the-art self-supervised visual representation learning approaches take advantage of data from multiple sensors by aligning the feature representations across views and/or modalities. In this work, we investigate how aligning representations affects the visual features obtained from cross-view and cross-modal contrastive learning on images and point clouds. On five real-world datasets and on five tasks, we train and evaluate 108 models based on four pretraining variations. We find that cross-modal representation alignment discards complementary visual information, such as color and texture, and instead emphasizes redundant depth cues. The depth cues obtained from pretraining improve downstream depth prediction performance. Also overall, cross-modal alignment leads to more robust encoders than pre-training by cross-view alignment, especially on depth prediction, instance segmentation, and object detection.