A Theoretical Characterization of Optimal Data Augmentations in Self-Supervised Learning

TL;DR

This paper uses kernel theory to analytically determine optimal data augmentations for self-supervised learning, revealing that augmentations need not resemble data and are heavily influenced by architecture.

Contribution

It provides a theoretical framework to design data augmentations in SSL, accounting for architecture effects and challenging the belief that augmentations must be data-like or diverse.

Findings

Augmentations do not need to resemble data to be effective.

Architecture significantly influences optimal augmentations.

Provides an algorithm for constructing augmentations based on target representations.

Abstract

Data augmentations play an important role in the recent success of self-supervised learning (SSL). While augmentations are commonly understood to encode invariances between different views into the learned representations, this interpretation overlooks the impact of the pretraining architecture and suggests that SSL would require diverse augmentations which resemble the data to work well. However, these assumptions do not align with empirical evidence, encouraging further theoretical understanding to guide the principled design of augmentations in new domains. To this end, we use kernel theory to derive analytical expressions for data augmentations that achieve desired target representations after pretraining. We consider non-contrastive and contrastive losses, namely VICReg, Barlow Twins and the Spectral Contrastive Loss, and provide an algorithm to construct such augmentations. Our analysis shows that augmentations need not be similar to the data to learn useful representations, nor be diverse, and that the architecture has a significant impact on the optimal augmentations.