An Augmentation Overlap Theory of Contrastive Learning

TL;DR

This paper introduces the augmentation overlap theory to explain contrastive learning's effectiveness, showing how aggressive data augmentations increase intra-class sample overlap, leading to better clustering and a new unsupervised evaluation metric.

Contribution

It proposes a new theoretical framework based on augmentation overlap, relaxing previous assumptions and deriving bounds for contrastive learning performance.

Findings

Supports the augmentation overlap theory with theoretical bounds.

Develops an unsupervised metric correlating with downstream performance.

Provides code for practical implementation.

Abstract

Recently, self-supervised contrastive learning has achieved great success on various tasks. However, its underlying working mechanism is yet unclear. In this paper, we first provide the tightest bounds based on the widely adopted assumption of conditional independence. Further, we relax the conditional independence assumption to a more practical assumption of augmentation overlap and derive the asymptotically closed bounds for the downstream performance. Our proposed augmentation overlap theory hinges on the insight that the support of different intra-class samples will become more overlapped under aggressive data augmentations, thus simply aligning the positive samples (augmented views of the same sample) could make contrastive learning cluster intra-class samples together. Moreover, from the newly derived augmentation overlap perspective, we develop an unsupervised metric for the representation evaluation of contrastive learning, which aligns well with the downstream performance almost without relying on additional modules. Code is available at https://github.com/PKU-ML/GARC.