On the Memorization Properties of Contrastive Learning

TL;DR

This paper investigates how contrastive learning, specifically SimCLR, memorizes training data and compares its memorization patterns to supervised learning and random labels, revealing similarities and differences in data complexity handling.

Contribution

It provides the first detailed analysis of memorization in contrastive learning, highlighting differences in how SimCLR learns objects and augmentations compared to supervised methods.

Findings

SimCLR's memorization of objects varies with complexity.

SimCLR's memorization patterns resemble those of random labels training.

Training objects and augmentations exhibit different complexity levels in SimCLR.

Abstract

Memorization studies of deep neural networks (DNNs) help to understand what patterns and how do DNNs learn, and motivate improvements to DNN training approaches. In this work, we investigate the memorization properties of SimCLR, a widely used contrastive self-supervised learning approach, and compare them to the memorization of supervised learning and random labels training. We find that both training objects and augmentations may have different complexity in the sense of how SimCLR learns them. Moreover, we show that SimCLR is similar to random labels training in terms of the distribution of training objects complexity.