Self-supervised Learning: Generative or Contrastive

TL;DR

This survey reviews recent self-supervised learning methods across vision, NLP, and graph domains, categorizing them into generative, contrastive, and generative-contrastive approaches, and discusses their theoretical foundations and future challenges.

Contribution

It provides a comprehensive categorization and analysis of existing self-supervised learning methods, highlighting their objectives, theoretical insights, and open problems.

Findings

Self-supervised learning improves representation learning without manual labels.

Contrastive and generative methods are the main categories of current approaches.

Theoretical analyses deepen understanding of how self-supervised learning functions.

Abstract

Deep supervised learning has achieved great success in the last decade. However, its deficiencies of dependence on manual labels and vulnerability to attacks have driven people to explore a better solution. As an alternative, self-supervised learning attracts many researchers for its soaring performance on representation learning in the last several years. Self-supervised representation learning leverages input data itself as supervision and benefits almost all types of downstream tasks. In this survey, we take a look into new self-supervised learning methods for representation in computer vision, natural language processing, and graph learning. We comprehensively review the existing empirical methods and summarize them into three main categories according to their objectives: generative, contrastive, and generative-contrastive (adversarial). We further investigate related theoretical analysis work to provide deeper thoughts on how self-supervised learning works. Finally, we briefly discuss open problems and future directions for self-supervised learning. An outline slide for the survey is provided.