Heterogeneous Contrastive Learning for Foundation Models and Beyond

TL;DR

This survey reviews heterogeneous contrastive learning techniques for foundation models, emphasizing their ability to handle view and task heterogeneity, and discusses open challenges and future research directions.

Contribution

It provides a comprehensive evaluation of current contrastive learning methods for foundation models across multiple domains and highlights future trends and open challenges.

Findings

Contrastive learning effectively handles view heterogeneity in foundation models.

Methods for task heterogeneity integrate contrastive loss with various training objectives.

The survey identifies key open challenges and future directions in heterogeneous contrastive learning.

Abstract

In the era of big data and Artificial Intelligence, an emerging paradigm is to utilize contrastive self-supervised learning to model large-scale heterogeneous data. Many existing foundation models benefit from the generalization capability of contrastive self-supervised learning by learning compact and high-quality representations without relying on any label information. Amidst the explosive advancements in foundation models across multiple domains, including natural language processing and computer vision, a thorough survey on heterogeneous contrastive learning for the foundation model is urgently needed. In response, this survey critically evaluates the current landscape of heterogeneous contrastive learning for foundation models, highlighting the open challenges and future trends of contrastive learning. In particular, we first present how the recent advanced contrastive learning-based methods deal with view heterogeneity and how contrastive learning is applied to train and fine-tune the multi-view foundation models. Then, we move to contrastive learning methods for task heterogeneity, including pretraining tasks and downstream tasks, and show how different tasks are combined with contrastive learning loss for different purposes. Finally, we conclude this survey by discussing the open challenges and shedding light on the future directions of contrastive learning.