Localizing Memorization in SSL Vision Encoders

TL;DR

This paper introduces methods to localize where memorization occurs within SSL vision encoders, revealing that memorization is widespread across layers and units, especially for outlier data, and varies by architecture.

Contribution

The paper proposes novel, task-independent metrics for localizing memorization in SSL encoders at layer and unit levels, applicable to various architectures and datasets.

Findings

Memorization increases with layer depth in SSL encoders.

A significant fraction of units exhibit high memorization, unlike supervised models.

Outlier data points cause higher memorization in layers and units.

Abstract

Recent work on studying memorization in self-supervised learning (SSL) suggests that even though SSL encoders are trained on millions of images, they still memorize individual data points. While effort has been put into characterizing the memorized data and linking encoder memorization to downstream utility, little is known about where the memorization happens inside SSL encoders. To close this gap, we propose two metrics for localizing memorization in SSL encoders on a per-layer (layermem) and per-unit basis (unitmem). Our localization methods are independent of the downstream task, do not require any label information, and can be performed in a forward pass. By localizing memorization in various encoder architectures (convolutional and transformer-based) trained on diverse datasets with contrastive and non-contrastive SSL frameworks, we find that (1) while SSL memorization increases with layer depth, highly memorizing units are distributed across the entire encoder, (2) a significant fraction of units in SSL encoders experiences surprisingly high memorization of individual data points, which is in contrast to models trained under supervision, (3) atypical (or outlier) data points cause much higher layer and unit memorization than standard data points, and (4) in vision transformers, most memorization happens in the fully-connected layers. Finally, we show that localizing memorization in SSL has the potential to improve fine-tuning and to inform pruning strategies.